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A. Baseline Data 


This appendix describes the participant baseline data that were measured at the time of program entry. 
Section A.1 describes the process by which the baseline data were collected. Section A.2 provides detail 
on the construction of the measures. Section A.3 discusses the prevalence of missing data and briefly 
describes the procedure used to multiply-impute missing data. 


A.1. DATA COLLECTION PROCESS 


The outcomes study enrolled participants from August 2016 through October 2017. This enrollment 
period captured participants who began their training programs in four cohorts: fall 2016, spring 2017, 
summer 2017, and fall 2017. Grantees with programs in the outcomes study provided staff who would 
oversee study enrollment and administer the baseline forms. The baseline forms included an informed 
consent form and a baseline information form. The research team trained and monitored grantee staff 
who administered the two forms to participants. 


Two site liaisons from the research team were assigned to each of the nine grantees. The site liaisons 
worked with grant leadership to identify at least one staff person at each participating college or campus 
to be trained to administer the baseline forms (called the “data collector”). The data collectors were 
typically program instructors or advisors. Site liaisons developed a data collection manual and training 
presentation for each grantee and training program. Site liaisons delivered the trainings virtually to grant 
staff. The manual and training covered all aspects of data collection, including enrollment and consent, 
content of the baseline information form, human subjects’ protections and privacy, data quality, and data 
security, storage, and transfer to the research team. Data collectors signed Individual Investigator 
Agreements to indicate they understood their responsibilities to ensure human subjects protections. 


The research team established outcome study eligibility requirements. Data collectors ensured that 
participants were eligible to participate in the study before beginning the data collection process. Across 
all outcome study programs, potential study participants needed to be at least 18 years old and be 
enrolled in an outcomes study program (e.g., not auditing the class). In addition, potential participants 
needed to meet any program-specific eligibility requirements, such as having a valid driver’s license for 
Cincinnati State’s CDL program. 


Site liaisons worked with grantees to determine at what point in a program study enrollment would occur. 
In general, the research team and grant staff aimed to enroll program participants the closest time to 
program start that made sense given program and staff constraints. Enrollment typically occurred on the 
first day of class or shortly thereafter, in person, as a group, in a classroom or other space on campus. As 
such, the majority of enrollments occurred at the beginning of each academic semester (including 
summer terms). A few of the very short-term programs, such as Cincinnati State’s CDL program, started 
more frequently. For these programs, study enrollment occurred more often in order to align with each 
start of a new training program session. 


During study enrollment, data collectors began by describing the study and consenting participants, using 
scripts and consent forms developed by the research team. Data collectors checked for participant 
comprehension of consent before asking them to sign two consent form copies (one for participants to 
retain and one for the research team’s records). Data collectors stressed that participation in the study 
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was voluntary and that program participation was not contingent on participating in the study. Grantee- 
level consent rates are provided in Exhibit A-1. 


Exhibit A-1. Participant Consents and Declines by Grantee 


Grantee | Consents (N) Declines (N) Consent Rate (%) 
Bossier Parish Community College 60 0 100.0 
Chaffey College 544 66 89.2 
tte Canaan and 115 57 66.9 
Delgado Community College 454 80 85.0 
Ivy Tech Community College 485 252 65.8 
Manchester Community College 362 119 75.3 
Miami Dade College 265 14 95.0 
South Central College 267 40 87.0 
Washburn University 215 17 92.7 
Total 2,767 645 81.1 


Consented individuals then completed the study’s baseline information form. They completed a paper 
version of the baseline information with pen/pencil and were able to ask data collectors questions about 
the form's content (e.g., to clarify the meaning of a question or word). While data collectors were trained 
to encourage study participants to answer every item on the baseline information form, they also 
explained that most items on the form were optional. After participants returned their completed forms to 
the data collectors, the data collectors reviewed the forms for completeness. Data collectors followed up 
with participants to address legibility issues or if there were a large number of unanswered items. After 
study enrollment, data collectors batched the forms by class and program and shipped the materials via 
FedEx to the research team. To ensure that the data were secure, site liaisons confirmed with grantee 
staff when each batch of forms was shipped and the number of forms included in each batch. 


Site liaisons provided ongoing support to grantee staff during the study enrollment period. Liaisons held 
planning calls with data collectors before each enrollment cycle to review the process to strategize about 
ways to improve enrollment rates and data quality, and to identify the specific day and time of baseline 
form administration. Liaisons also trained new staff members as needed (e.g., following staff turnover) 
and had debriefing discussions with data collectors at the end of each enrollment cycle. In addition, study 
leadership monitored enrollment rates and sample sizes over the study enrollment period. The research 
team identified programs with lower than desired enrollment rates and provided additional support to 
increase rates in subsequent enrollment cycles. 


Once the research team received each FedEx package, they logged the forms, assigning each participant 
an anonymized unique identifier. The research team built an online data entry system to input data from 
the baseline information forms. Research assistants and temporary staff, who had been trained in data 
security procedures, entered the data into the online system. In order to minimize the risk of data entry 
error, the team entered the data from the forms a second time into a separate database. The research 
team then manually checked and reconciled any inconsistencies in data entry. 
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A.2. DETAILS ON BASELINE MEASURES 


The research team used data collected from the Baseline Information Form (BIF) to construct a variety of 
baseline measures. This report uses the baseline measures for several purposes, including: the 
description of participant characteristics in Chapter 4; definitions of subgroups in Chapter 5; predictors of 
service receipt in Chapter 6; and predictors of program-specific outcomes in Chapter 7. Exhibit A-2 
describes the definition and construction of the baseline measures. 


Exhibit A-2. Description of Baseline Measures 


Baseline 
Measure 


Age at program 
entry 


Description and Operationalization 


Categorical measure of participant age at the time of study 
enrollment. Derived from date of birth and the date of enrollment 
into the study. Grouped into four categories (age 20 or less; age 21 
to 24; age 25 to 34; age 35 or older). 


APPENDIX A 


Baseline Information 


Form Items 


Question 7 and 
enrollment date from 
informed consent form 


Sex (male; 
female) 


Binary indicators for whether participant was male or female. 


Question 6 


Race/ethnicity 


Veteran status 


Categorical indicator of participant race and ethnicity. Based on two 
questions: 
“Are you of Spanish, Hispanic, or Latino origin? 
“Do you consider yourself to be (select one or more): 
1) American Indian or Alaskan Native 
2) Asian 
3) Black or African-American 
4) Native Hawaiian or other Pacific Islander 
5) White” 


Responses were grouped into five categories: 
1) Hispanic (any race) 
2) White (Non-Hispanic) 
3) Black (Non-Hispanic) 
4) Asian (Non-Hispanic) 
5) Other, including multi-race (Non-Hispanic) 


Binary indicator for whether the participant had ever been active 


duty in the military. 


Questions 9 and 10 


Question 16 


Living with 
spouse or 
partner 


Binary indicator for whether the participant was currently married or 
living with a partner. Based on question “What is your current 
marital status? 

1) Married 

2) Living with partner 

3) Widowed 

4) Divorced/separated 

5) Never married” 


Responses 1 and 2 were classified as living with a spouse or 
partner; responses 3, 4, and 5 were classified as not living with a 
spouse or partner. 


Question 8 
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Baseline 
Measure 


Living with 
children 


Description and Operationalization 


Binary Indicator for whether the participant was living with any of 
their own children age 17 or younger. Based on question “How 
many of your children (17 years or younger) currently live in your 
household?” 


Response of 1 or more was classified as living with children; 
response of 0 was classified as not living with children 
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Form Items 


Question 13 


Educational 
attainment 


Categorical indicator of the participant's highest level of education. 
Based on question “What is the highest degree or level of school 
you have completed (select only one): 

1) No formal education 

2) 12 grade or less, no diploma 

3) High school graduate 

4) GED 

5) Technical, trade, or vocational degree 

6) Some college credit, but no degree 

7) Associate’s degree 

8) Bachelor’s degree 

9) Master’s degree or higher” 


If a respondent selected more than one option, the research team 
used the highest level. Responses were grouped into six categories: 
1) Less than high school (responses 1 and 2) 
2) High school or GED (responses 3 and 4) 
3) Technical, trade, or vocational degree (response 5) 
4) Some college credit, but no degree (response 6) 
5) Associate’s degree (response 7) 
6) Bachelor’s degree or higher (response 8) 


Question 17 


Employment 
status 


Categorical indicator of the participant's employment status. Based 
on question “What is your current employment status? 
1) Iam-currently working at one or more jobs or 
businesses. 
2) Iam not currently working, but I have worked at one 
or more jobs or businesses during the last 12 months. 
3) It has been longer than 12 months since I last 
worked at a job or business. 
4) I have never been employed. 


Question 23 
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Baseline 
Measure 


Current/most 
recent industry 


Description and Operationalization 


Categorical indicator of the industry in which the participant was 
currently or most recently employed. Based on question “Which of 
the following industries best matches the business of your 
current/last employer?” 


Participants could choose among 20 industries, which were 
aggregated to the following eight categories: 

1) Manufacturing 

2) Construction 

3) Transportation and warehousing 

4) Accommodation and Food Services 

5) Retail trade 

6) Health care and social assistance 

7) Professional, scientific, and technical services 

8) Other services 


Responses are reported separately for those who were employed at 
program enrollment, and for those who were not employed but did 
have some work history. 
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Form Items 


Question 24 


Hourly wage 


Continuous variable that measures the hourly wage the participant 
earned at their current or most recent job. 


For participants employed at program entry, based on response to 
question “How much do you earn per hour at your main job, before 
taxes and other deductions?” 


For participants not employed at program entry, but who had 
worked during the past 12 months, based on response to question 
“When you were working, how much did you earn per hour at your 
main job?” 


Questions 23d and 23f 


Expect to be 
working for pay 


Indicator for whether the participant expects to be working for pay 
in the next few months. 


Question 26 


Expect to be 


Indicator for whether the participant expects to be working full-time 


working full- (35 or more hours per week) or part-time (1 to 34 hours. Based on Question 26a 
time or part- question “How many hours a week do you expect to be working in 
time the next few months?” 
Categorical indicator of primary reason for training. Based on 
question “What is the most important reason you decided to enroll 
in this program? 
1) Find work 
R f ; 
(anno 2) Career change Question 27 
3) Career advancement 
4) Educational advancement 
5) Personal reasons 
6) Other” 
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Years of 
experience in 
target industry 


Family income 


Description and Operationalization 


Categorical indicator of years of experience in target industry. Based 
on question “Please enter the number of years’ (and/or months’) 
experience you have in the industry for which you are applying for 
training.” 


Responses were classified into three categories: 
1) No experience (0 months and 0 years) 
2) Less than 1 year of experience (1 to 11 months) 
3) One year or more (12 months or more, or 1 year or 
more) 


Categorical indicator of family income. Based on question “Please 


mark that best matches your total family income over the last 12 
months, including earnings, pensions, public assistance, alimony, 
child support, Veteran’s payments, etc., before deductions for taxes, 
bonds, dues, or other items. 

1) $0 

2) $1-$9,999 

3) $10,000-$14,999 

4) $15,000-$19,999 

5) $20,000-$29,999 

6) $30,000 or over” 


Responses were grouped into three categories: 
1) Less than $15,000 (responses 1, 2, and 3) 
2) $15,000 to $29,999 (responses 4 and 5) 
3) $30,000 or more (response 6) 


APPENDIX A 


Baseline Information 
Form Items 


Question 28 


Question 25 


Continuous measure of the number of people in household. Based 
on questions “How many of your children (17 years or younger) 


Number of currently live in your household?” and “Not including yourself, how . 
people in many adults (18 years or older) currently live in your household?” Questions 13 and 14 
household 
Used in the construction of the poverty measure. 
Indicator for whether the household was below the federal poverty 
level. Constructed from exact family income and number of people 
Poverty status in household, using 2017 federal poverty guidelines Questions 13, 14, and 
(https://aspe.hhs.gov/2017-poverty-quidelines). For example, the 25 
poverty level for a family of three was $20,420. See Section A.3 for 
discussion of imputation of exact family income. 
Binary indicators for whether the participant was currently receiving 
Public benefits from each of the following sources: 
alee pas 1) Trade Readjustment Allowances (TRA) Questions 19, 20, 21, 
receipt 2) Temporary Assistance for Needy Families (TANF) and 22 
3) Supplemental Nutrition Assistance Program (SNAP) 
4) Unemployment Insurance 
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A.3. IMPUTATION OF MISSING BASELINE MEASURES 


As is typical in any survey data collection, some participants did not respond to certain questions on the 
BIF form. As shown in Exhibit A-3, item nonresponse ranged from less than 5 percent for demographic 
characteristics to over 15 percent for questions related to employment, earnings, and income. 


Exhibit A-3. Missing Data Rates for Baseline Measures Prior to Imputation 


Baseline Measure Missing Data Rate (%) 


Age at program entry 2.2 
Sex 3.1 
Ethnicity 3.4 
Race 4.4 
Veteran Status 4.3 
Living with spouse or partner 3.5 
Living with children 4.4 
Educational attainment 4.3 
Number of children in household 4.4 
Number of adults in household 5.5 
Employment status 12.0 
Current/most recent industry 15.6 
Hourly wage 14.2 
Expect to be working for pay 11.0 
Expect to be working full-time or part-time 15.1 
Reason for training 8.9 
Years of experience in target industry 10.2 
Family income (categorical) 15.8 
Poverty status 16.2 
Trade Readjustment Allowances (TRA) receipt 5.1 
Temporary Assistance for Needy Families (TANF) receipt 5.3 
Supplemental Nutrition Assistance Program (SNAP) receipt 4.8 
Unemployment Insurance receipt 4.7 


Source: Baseline Information Form 
Note: Sample includes all study participants (N = 2,767) 


The research team used multiple imputation (Rubin 1987, 1996) to address this item nonresponse. In 
order to include information from both the BIF and follow-up survey in the imputation models, a set of 
core baseline and follow-up survey outcomes were imputed jointly. Since both baseline and follow-up 
survey variables were imputed together imputation model, and the procedure was similar for the two 
types of data, the full discussion of the multiple imputation appears in Appendix B.3, after the discussion 
of follow-up survey measures. 


A.4. INFERENCES ABOUT THE PARTICIPANTS FOR PROGRAMS 
LIKE THOSE FUNDED BY TAACCCT 


The 34 programs studied in this report served a particular set of participants, but the profile of the 
student body varied considerably across the programs. If there were another round of TAACCCT grants, 
the potential participants would doubtless differ in many respects. Much would depend on the labor 
market at the time. As the novel coronavirus pandemic of 2020 has demonstrated, labor market 
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conditions can change very quickly. The programs in this study were operating near the end of an 
extraordinarily long expansion that, despite near record low unemployment, had left positions in some 
industries, such as advanced manufacturing, healthcare, and IT unfilled. Additionally, the nature of the 
economy necessitates that some workers need to be retrained. Once the pandemic has passed and the 
people who are most easily employed are back at work, the nation will eventually reach the point again 
at which there is general prosperity but pockets of people face chronic unemployment or 
underemployment unless they are re-trained. Future program or grant planners may find it useful to have 
some idea of what types of participants would avail themselves of the new prorgams at that future time. 
While that future planner could simply take the observed averages from this study, they would be better 
served to consider the condidence intervals presented in Exhibit G.1 of Appendix G. 


For example, this study found that 59.5 percent of participants expected at study intake to enroll full 
time. However, this percentage varied widely across programs. As a result, the 95 percent confidence 
interval on this percentage runs from 48.4 to 70.6 percent. This confidence interval is strictly valid only if 
the 34 programs can be viewed as a simple random sample the universe of programs that could be 
funded in the future. While this assumption is clearly not tenable, future planners would almost certainly 
do a better job of planning if they planned for a full-time enrollment rate between 50 and 70 percent, 
rather than assuming either (1) that it would be exactly 59.5 percent again or (2) that all values between 
0 and 100 percent are equally plausible. 


The confidence intervals of Section G.1 were calculated using the SAS procedure SURVEYMEANS with an 
adjustment for multiple imputation calculated with another SAS procedure, MIANALYZE. With 
SURVEYMEANS, the team estimated the “full-sample” variance for a single one of the multiple 
imputations. It was run with CLUSTER=PROGRAM specification of SURVEYMEANS so as to reflect the 
extra uncertainty caused by the clustered nature of the sample. The nonresponse weights were also 


used. Let the variance estimated by SURVEYMEANS for the r-th imputed dataset be denoted as Q,,,. and 


let the corresponding mean (based on that imputation) be / er): With MIANALYZE, the cross-imputation 


variance was calculated as: 


5 


1 = 
a tee ~ I. i : 


U,= 


where I, is the average estimated mean across the five multiple imputations. 
Finally, the total variance of the estimated mean (such as the mean proportion of students expecting to 


study full-time) is computed using the Rubin‘s rules to combine the within- and between-imputation 
variances and produce a consistent estimate of the overall variance: 


12 6 
if 7 oane tu, ; 
Dae 5 
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B. Follow-up Survey Data 


This appendix describes the 12-month follow-up survey data collected from study participants. 

Section B.1 describes the data collection process. Section B.2 provides detail on the construction of the 
outcomes from the survey data. Section B.3 discusses the prevalence of item nonresponse and the 
procedure used to multiply-impute missing data. Section B.4 describes the study’s approach to survey 
nonresponse. 


B.1. DATA COLLECTION PROCESS 


The 12-month follow-up survey was trimodal: participants first had the opportunity to complete the 
survey via web, followed by phone, and then in-person. The research team administered the survey to 
2,211 of the 2,767 study participants, yielding an 80 percent response rate. Survey administration 
occurred between September 28, 2017 and April 1, 2019 in both English and Spanish. 


After participants enrolled in the study, the research team stayed in contact with them to keep contact 
information updated and thus to increase the likelihood that they would receive and complete the follow- 
up survey. First, participants received via mail a package welcoming them to the study shortly after 
program entry. Participants then received tracking emails approximately three and six months after 
program entry and a tracking letter approximately nine months after program entry. Approximately 12 
months after program entry, participants received a prenotification letter and email telling them that 
survey administration was imminent and that they could complete the survey online. 


The survey collected information on training receipt, perceived training quality, and educational progress; 
training-related supports; employment characteristics; and income and public assistance benefits receipt.! 
Most of the questions came from previous data collection efforts conducted by Abt Associates, such as 
the short-term and three-year follow up surveys for the first round of the Health Profession Opportunity 
Grants (HPOG) Program evaluation (HHS), the Pathways for Advancing Careers and Education (PACE) 
evaluation (HHS), the Green Jobs and Health Care impact survey (DOL), and the Job Search Assistance 
Strategies impact survey (HHS). The research team pretested the questionnaire in November 2016 ona 
convenience sample of nine participants who had enrolled in Information Technology programs at Ivy 
Tech College in fall 2015. Once the questionnaire was finalized, the research team programmed it in 
ConfirmIt for Computer Aided Web Interviewing and Computer Aided Personal Interviewing 
administration. 


As described in Appendix A, the sample was recruited in four cohorts, which corresponded to the start of 
each academic term: fall 2016, spring 2017, summer 2017, and fall 2017. Exhibit B-1 shows the launch 
and closing dates of the follow-up survey by cohort. For each cohort, the survey was launched 
approximately one year after enrollment into the study. 


Exhibit B-1. Follow-up Survey Launch and Closing Dates by Cohort 


Cohort Recruitment Term | Survey Launch Date Survey Closing Date 
Fall 2016 September 2017 June 2018 
Spring 2017 March 2018 January 2019 


1 The questionnaire is available from the authors upon request. 
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Summer 2017 June 2018 March 2019 
Fall 2017 September 2018 April 2019 


When the survey launched for each cohort, participants first received an email invitation to complete the 
web-based survey. The research team emailed participants weekly reminders to complete the survey. 
After approximately one month of web-based survey administration, field interviewers from the research 
team began to contact non-respondents by phone. After about one month of telephone interviewing 
attempts, field interviewers began to visit non-respondents in person. Participants received a $25 Visa 
debit card as a thank you for completing the survey. 


Field interviewers, who conducted both phone and in-person interviews, were trained on survey 
administration in August 2017. The same training was delivered to interviewers who joined the team at a 
later date because of staff turnover. For quality assurance purposes, field interviewers recorded their 
interviews with participants (once participants consented to being recorded). The research team reviewed 
10 percent of each field interviewer's audio recordings to confirm that survey protocols were being 
followed. 


Exhibit B-2 summarizes the final disposition of cases for the survey. 


Exhibit B-2. Final Survey Dispositions 


Disposition Number of Participants 
Enrolled in study 2,/67 
Completed survey 2,211 
Refused or broke off 126 
Deceased or incarcerated 15 


Ineligible for survey due to insufficient identifying 
information (no date of birth, SSN, or contact information) 


Unlocatable, unavailable, or other non-refusal 374 


41 


B.2. OUTCOMES BASED ON FOLLOW-UP SURVEY DATA 


Exhibit B-3 provides a detailed description of all outcomes created from the follow-up survey. 


Abt Associates Round 4 Early Outcomes Study Report December 2020 1] B-2 
Technical Appendices 


APPENDIX B 


Exhibit B-3. Description of Follow-up Survey Measures 


Survey Measure 


Training Duration Outcomes 


Total months of training 


Definition 


Calculated as the number of months between the month of enrollment and either the month 
the participant finished classes (if not still enrolled at follow-up) or the interview month (if 
still enrolled at follow-up). 


For example, someone who enrolled in January and finished classes in March was credited 
with two months of training. 


ito) Co) WENT 9) 
Survey Items 


Bi, B2 


Full-time equivalent (FTE) months of 
training 


Number of full-time equivalent months enrolled in training, where 12 hours per week or 
more is considered full-time. Calculated as follows: 
e If participant spent 12 or more hours per week in training, then full-time 
equivalent months of training is equal to total months of training. 
e If participant spent 1 to 11 hours per week in training, then full-time 
equivalent months of training is equal to (total months of training) x (hours 


per week in training / 12). 


Bi, B2, B8 


working in groups 


Completed at least 6 FTE months of training | Binary indicator for whether the participant completed at least 6 FTE months of training Bi, B2, B8 
Service Receipt Outcomes 

Ever took a course focusing on study skills, Ever took a course during training that focused on study skills, workplace skills, or general B16 
workplace skills, or general life skills life skills 

Ever took life skills course focusing on: Ever took a course during training that focused a great deal of attention on career planning BI7A 
career planning 

Ever took life skills course focusing on: Ever took a course during training that focused a great deal of attention on study skills B17B 
study skills 

Ever took life skills course focusing on: job Ever took a course during training that focused a great deal of attention on finding a job or BI7C 
search moving to a different job 

Ever took life skills course focusing on: Ever took a course during training that focused a great deal of attention on critical thinking B17D 
critical thinking and problem solving skills and problem-solving skills 

Ever took life skills course focusing on: Ever took a course during training that focused a great deal of attention on finding help with 

oat eo We B17E 
finding help with life problems problems at school, work or home 

Type of life skills course: financial aid for Ever took a course during training that focused a great deal of attention on finding and BI7F 
school applying for financial aid for school 

Ever took life skills course focusing on: time | Ever took a course during training that focused a great deal of attention on managing time B17G 
management effectively 

Ever took life skills course focusing on: Ever took a course during training that focused a great deal of attention on working in B17H 


groups 
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Definition 


ito) Co) WEIN 8) 
Survey Items 


Ever took life skills course focusing on: Ever took a course during training that focused a great deal of attention on communicating B17I 
communicating well well 
Ever took life skills course focusing on: Ever took a course during training that focused a great deal of attention on managing stress B17] 
managing stress and anger and anger 
Ever took life skills course focusing on: Ever took a course during training that focused a great deal of attention on staying B17K 
staying motivated motivated 
Ever took life skills course focusing on: Ever took a course during training that focused a great deal of attention on acting B17L 
acting professionally professionally 
Ever took life skills course focusing on: Ever took a course during training that focused a great deal of attention on managing money B17M 
managing finances and personal finances 
Ever took life skills course focusing on: Ever took a course during training that focused a great deal of attention on handling B17N 
handling family responsibilities parenting and other family responsibilities 
Offered a work study job as part of studies Participant stated that they were offered a work study job as part of their studies. B18A 
Offered clinical experience or practicum as Participant stated that they were offered clinical experience or a practicum as part of their B18B 
part of studies studies. 
Offered arranged visits from Participant stated that they were offered arranged visits from or to learn about an individual 
employer/learning about employers as part employer as part of their studies. Bi8C 
of studies 
Offered class taught by instructors from Participant stated that they were offered a class taught by instructors from a local employer 
local employer/class offered on-site at local | as part of their studies. Bi8D 
employer as part of studies 
Offered an apprenticeship as part of studies | Participant stated that they were offered an apprenticeship as part of their studies. B18E 
Offered an internship as part of studies Participant stated that they were offered an internship as part of their studies. B18F 
Offered other work experience as part of Participant stated that they were offered other work experience as part of their studies. B18G 
studies 
This composite measure indicates whether participants were offered one or more B18A, B18B 
Offered opportunity for work study job or opportunities for a work study job, clinical experience or practicum, apprenticeship, B18E. BIS. 
internship internship, or other work experience as part of their studies. Arranged visits from employers B18G . 
and classes taught by instructors from local employers were not included in this measure. 
Received academic advising Participants were asked whether they received academic advising, either through the school C4A 
or through a referral from the school to another organization 
Received financial aid advising Participants were asked whether they received financial aid advising, either through the C4B 
school or through a referral from the school to another organization 
Received career counseling Participants were asked whether they received career counseling, either through the school c4c 
or through a referral from the school to another organization 
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Definition 


ito) Co) WENT 9) 
Survey Items 


F : : Participants were asked whether they received job search or placement assistance, either 
el ea through the school or through a referral from the school to another organization saa 
: : es Participants were asked whether they received help creating or editing their resume, either 
ee eI eer at ne orcad (une through the school or through a referral from the school to another organization aus 
Participants who reported that they received job search or placement assistance (C4D=1) 
Received help looking for a job were asked whether they received help looking for a job, either through the school or C5B 
through a referral from the school to another organization 
Received Halouicina webubased Search Participants who reported that they received job search or placement assistance (C4D=1) 
: p 9 were asked whether they received help using web-based job search engines, either through C5C 
engines ponent 
the school or through a referral from the school to another organization 
Participants who reported that they received job search or placement assistance (C4D=1) 
Received help finding specific job leads were asked whether they received help finding specific job leads, either through the school C5D 
or through a referral from the school to another organization 
Participants who reported that they received job search or placement assistance (C4D=1) 
Received help filling out job applications were asked whether they received help filling out job applications, either through the school C5E 
or through a referral from the school to another organization 
Participants who reported that they received job search or placement assistance (C4D=1) 
Received practicing for job interviews were asked whether they received help practicing for job interviews, either through the C5F 
school or through a referral from the school to another organization 
Source of funds for training: own earnings Participant reported that their own earnings or savings were used to help pay for program CIA 
or savings or those of a spouse expenses 
Source of funds for training: loans in own Participant reported that loans in their or a family member’s name were used to help pay for C1B 
name or name of a family member program expenses 
Source of funds for training: a parent or Participant reported that financial help from a parent or other family member was used to CIC 
other family member help pay for program expenses 
Source of funds for training: grant from the | Participant reported that a government grant was used to help pay for program expenses C1D 
government 
Source of funds for training: used TRA Participant reported that TRA/TAA benefits were used to help pay for program expenses CIE 
benefits to pay for training costs 
Source of funds for training: Veteran’s Participant reported that Veteran's benefits were used to help pay for program expenses CIF 
benefits 
Source of funds for training: scholarship Participant reported that a scholarship was used to help pay for program expenses C1iG 
Source of funds for training: financial Participant reported that financial support from their employer was used to help pay for CiH 
support from employer program expenses 
Source of funds for training: other funding Participant reported that another funding source was used to help pay for program expenses CII 
source 
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Survey Measure Definition oleyaue 
Survey Items 
Program satisfaction Categorical indicator of whether the participant was “very satisfied,” “somewhat satisfied,” or B19 
not satisfied 
Training Progress Outcomes 
Finished classes Participant finished the required classes in their program of study Bi 
Left without finishing classes Participant left program without completing the required classes in their program of study Bi 
Still enrolled Participant was still enrolled in the required classes in their program of study at follow-up Bi 
Program completion ise Ie eto die Coe be eieiotameine tues B3 
Started additional training eniala completed the program, received a credential, and started an additional training BIS 
Plan to return to college Participant planned to return to college at some point in the future B1i5A, B15B 
Number of months until planning to return For participants who planned to return to college, this measures the number of months until B15C 
to college they planned to return 
Earned any college credits Participant earned any college credits as part of their program BO 
Number of college credits Total college credits earned as part of the program. Excludes credits transferred from other BOA 
institutions and credits for prior learning 
Employment, Earnings, and Income Outcomes 
Currently employed after finishing/leaving Participant currently working at a job for pay after finishing or leaving the program. Missing D1 
program for those still enrolled in program. 
Ever employed after finishing/leaving Participant reported any paid work since finishing or leaving the program. Missing for those D1. D2 
program still enrolled in program. j 
Ever employed after finishing/leaving For participants who had any paid work after finishing or leaving the program, measures 
program and job was in occupation related whether that job was “closely related” to their training program. Missing for those still D4 
to training program enrolled in program. 
Currently employed after finishing/leaving For participants who were currently employed after finishing or leaving the program, 
program and job was in occupation related measures whether that job was “closely related” to their training program. Missing for those D4 
to training program still enrolled in program. 
Ever employed after completing program For participants who had any paid work after completing the program, measures whether 
and job was in occupation related to training | that job was “closely related” to their training program. Missing for those still enrolled in D4 
program program. 
Currently employed after completing For participants who were currently employed after completing the program, measures 
program and job was in occupation related whether that job was “closely related” to their training program. Missing for those still D4 
to training program enrolled in program. 
Currently employed full-time after Participants who were currently employed after finishing or leaving the program and worked D6 
finishing/leaving program at least 35 hours per week. Missing for those still enrolled in program. 
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Definition 


it) Co) WENT 9) 
Survey Items 


Currently employed part-time after Participants who were currently employed after finishing or leaving the program and worked D6 
finishing/leaving program less than 35 hours per week. Missing for those still enrolled in program. 
Indicator for whether participants who were currently working part-time after finishing or 
Underemployed leaving the program would have preferred to work full-time. Missing for those still enrolled in D7 
program. 
Reason for wanting part-time work: child Participant reported child care problems as the main reason they did not want to work full- D8 
care problems time (defined for those working part-time who did not want to work full-time) 
: vr . Participant reported family or personal obligations (other than child care problems) as the 
eaSon fot Wanulng part time work Olier main reason they did not want to work full-time (defined for those working part-time who did D8 
family or personal obligations P 
not want to work full-time) 
Reason for wanting part-time work: health Participant reported health or medical limitations as the main reason they did not want to D8 
or medical limitations work full-time (defined for those working part-time who did not want to work full-time) 
: . ; Participant reported that retirement or social security limit on earning as the main reason 
Reason for wanting part-time work: retired : hs é : : : 
or Social Security limit on earnings they did not want to work full-time (defined for those working part-time who did not want to D8 
work full-time) 
. . ae Participant reported satisfaction with income from part-time hours as the main reason they 
Reason for wanting part-time work: satisfied : ve . : a : 
with income from part-time hours rea to work full-time (defined for those working part-time who did not want to work D8 
Currently employed in job with health Participant was currently employed in a job that offered health insurance and paid sick days 
rahe apace ieee ae D5A, D5D 
insurance and paid sick days since finishing or leaving the program. Missing for those still enrolled in program. 
Currently employed in job with health Participant was currently employed in a job that offered health insurance after finishing or DSA 
insurance leaving the program. Missing for those still enrolled in program. 
Currently employed in job with paid vacation Participant was currently employed ina job that offered paid vacation days after finishing or DSB 
leaving the program. Missing for those still enrolled in program. 
Currently employed in job with paid holidays Participant was currently employed ina job that offered paid holidays after finishing or DSC 
leaving the program. Missing for those still enrolled in program. 
Currently employed in job with paid sick Participant was currently employed in a job that offered paid sick days after finishing or DS5D 
days leaving the program. Missing for those still enrolled in program. 
Currently employed in job with retirement or | Participant was currently employed in a job that offered retirement or pension benefits after DSE 
pension benefits finishing or leaving the program. Missing for those still enrolled in program. 
Household receipt of TANF Anyone in participant's household received income or benefits from TANF (Temporary F1 
Assistance for Needy Families) in the prior month 
Household receipt of SNAP Anyone in participant's household received income or benefits from SNAP (Supplemental E2 
Nutrition Assistance Program) in the prior month 
Household receipt of TRA Anyone in participant’s household received income or benefits from TRA (Trade F3 
Readjustment Allowances) in the prior month 
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Definition 


ito) Co) WENT 9) 


Household receipt of other federal support 


Anyone in participant's household received income or benefits from federally funded 
programs other than TANF, SNAP or TRA in the prior month 


Survey Items 


E4 


Household receipt of any public assistance 
benefits 


This composite measure indicates whether the participant’s household received income or 
benefits from TANF, SNAP, TRA or another federally funded program in the prior month. 


E1, E2, E3, E4 


Household income 


Total household income over the prior 12 months, including “earnings, pensions, public 
assistance, alimony, child support, Veterans’ payments, etc., before deductions for taxes, 
bonds, dues, or other items.” Reported as either a continuous measure (E5) or categorical 
(E5A), with the following response categories: 

1) $0 

2) $1-$9,999 

3) $10,000-$14,999 

4) $15,000-$19,999 

5) $20,000-$29,999 

6) $30,000 or over” 
Exact income amounts imputed for those participants who only indicated a category. 


E5, ESA 


Number of people living in household 


Total number of people currently living in the household, including adults and own children 


E6, E7 


Indicator for whether the household was below the federal poverty level. Constructed from 
exact family income and number of people in household, using 2017 federal poverty 


Poverty status guidelines (https://aspe.hhs.gov/2017-poverty-quidelines). For example, the poverty level for | E5, E5A, E6, E7 
a family of three was $20,420. See Section B.3 for discussion of imputation of exact 
household income. 
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B.3. IMPUTATION OF MISSING FOLLOW-UP MEASURES 


Survey respondents had the option to decline to respond to any question on the follow-up survey, so 
each measure has some amount of missing data. Exhibit B-4 shows item nonresponse rates for selected 
follow-up survey measures. Missing rates ranged from about 2 to 4 percent for measures related to 
service receipt and training outcomes; 3 to 8 percent for questions related to employment and benefits 
receipt; and over 18 percent for household income. 


Exhibit B-4. Missing Data Rates for Selected Follow-up Survey Measures Prior to Imputation 


Follow-up Survey Measure ilssing Data Rate 


(%) 
Finished classes / left without finishing classes / still enrolled in classes 1.8 
Completed program and received target credential 2.9 
Received academic advising 3.0 
Received financial aid advising 3.4 
Received job search or placement assistance 3.2 
Offered opportunity for work study job or internship 3.7 
Ever took a course focusing on study skills, workplace skills, or general life skills 2.6 
Employed after finishing or leaving program 3.1 
Employed after finishing or leaving program in a job closely related to training 4.1 
Employed in a job that offers health insurance 5.6 
Employed in a job that offers paid sick leave 7.6 
Household receipt of any public assistance benefits 5.3 
Household income, either exact or categorical 18.2 


Source: 12-month follow-up survey. 
Note: Sample includes all survey respondents participants (N = 2,211) 


B.3.1 APPROACH TO IMPUTATION 


As noted, to address item nonresponse in both the baseline and follow-up survey data, the research team 
used standard multiple imputation procedures (Rubin 1987, 1996). Multiple imputation involves using a 
regression model to fill in missing values. This is done multiple times (hence “multiple” imputation), 
resulting in a set of 12 imputed values; the team then averaged across the imputed values to produce a 
final value and standard error. 


The imputation included a mix of iterated chain equations (ICE), also referred to as fully conditional 
specification (FCS), and univariate imputation (van Buuren 2018). Each variable that has missing values 
was assigned an imputation regression-like model. For most types of outcomes, a generalized linear 
model type is applicable. Binary outcomes were imputed using logit models; multinomial outcomes were 
imputed using multinomial logit models; conditional outcomes were imputed using predictive mean 
matching based on linear regression models; and interval-valued measures were imputed using interval 
regression. 


In order to include information from both baseline and follow-up survey data in the imputation models, a 
set of core of demographic variables and follow-up survey outcomes were imputed jointly by FCS. The 
remaining variables received univariate imputations, conditionally on that core of imputed variables. 
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B.3.2 VARIABLE BLOCKING 


To keep models concise and convergent, variables were imputed in two steps—primary FCS imputation, 
and secondary marginal imputation. 


Primary FCS Imputation 


First, a set of key baseline and follow-up measures was jointly imputed. These variables include: 


e Baseline measures: 


— Gender 

— Age category 

— Race and ethnicity 

— Marital status 

— Education 

— Citizenship 

— Language spoken at home 
— Family income category 

— Employment history 


e Follow-up survey measures: 


— Finished classes 

— Employed in a job related to training (among those not still enrolled in training) 
— Receipt of public assistance benefits 

— Family income 


This set of variables was simultaneously imputed using the ICE/FCS approach. All of the variables were 
cross-utilized in each other's regressions. To control for the local effects, grantee indicator variables were 
used as predictors in the demographic variable regressions, and program indicator variables were used in 
the follow-up survey outcome regressions. Also, an indicator variable for whether the programs aimed to 
train participants for occupations in goods-producing industries (Such as manufacturing, HVAC, industrial 
automation, pre-engineering, machining, or mechatronics) was used in all regressions. Finally, the 
program length was used as a predictor of completion of classes and post-program income, and some 
demographic interactions (age by gender, age by race/ethnicity, and gender by age) were used in the 
demographic variable and income regressions. Thus, each imputation regression had between 20 and 

40 predictor variables. 


Twelve imputed datasets were created for the full baseline sample. For reporting purposes, the imputed 
data for the follow-up survey variables were later subset to the appropriate sample of survey 
respondents. 


The type of statistical software used to implement multiple imputation differed due to the availability of 
software in different computing environments. For analysis of survey-based outcomes in Chapters 4 and 
5, the baseline measures and survey outcomes were imputed with Stata 16 statistical software. The 
analysis of earnings data was run on different analytic platforms (see Appendix C for information on 
National Directory of New Hires Unemployment Insurance wage data). The analyses underlying Chapter 6 
were performed on a DOL SAS server and used the SAS/MI/FCS procedure. For this analysis, only five 
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multiple imputations were used. The analyses underlying Chapter 7 were performed on a DOLR 
container, and used R version of FCS called MICE (Multivariate Imputation by Chained Equations) to 
implement multiple imputation. 


Secondary Marginal Imputation 


Once the set of key baseline and follow-up measures was imputed using the ICE/FCS procedure 
described in the previous section, all other variables were imputed using marginal regressions. For each 
missing variable, a regression model of the appropriate type (logit, multinomial logit, ordinal logit, 
predictive mean matching) was used to create the necessary imputations. Imputations were made 
conditionally on screener variables where appropriate; these screener variables would have been imputed 
as needed, as well. For these single equation models, program or grantee indicators were used, as 
needed. 


For very sparse data, models were simplified. For example, to impute the missing industry of the current 
or previous occupation in the baseline data, a multinomial model with imputation variables and program 
indicators were used for the most populated industries (manufacturing, retail, food service, construction, 
health care, transportation) vs. all others combined, using imputed variables and program indicators as 
predictors. The other remaining smaller industries were imputed unconditionally using the observed 
prevalence rates, without any attempt to control for demographics or program/grantee effects. In 
addition, in order to impute detailed race, a model had to be used that only contained gender, marital 
status, education, and age. 


B.3.3. VARIABLE TYPES 


The imputation regression model differed by the type of variable (binary, multinomial, continuous, or 
interval). 


Binary and Multinomial Variables 


Most variables were binary (0/1), and were imputed using logit models. Multinomial variables with more 
than two categories were imputed with multinomial logit models. The multinomial set of multinomial 
variables includes: 


e Marital status 

e Race 

e Employment history (never employed; employed in the past; currently employed at follow-up) 
e Housing occupancy status (own; rent; live rent-free) 


e Most important motive for enrollment (6 categories: find work, career change, career advancement, 
education advancement, personal, other) 


e Industry of current or former employment 
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Continuous Variables 


The continuous variables included: 


e Number of adults in the household at baseline and at follow-up 

e Number of children in the household at baseline and at follow-up 
e Hourly wage at baseline at current or previous job 

e Number of college credits or transfer credits earned 

e Actual weeks in training 


e Hours per week worked, hours per week in training 


Given the irregular distribution of these variables (e.g., college credits spiking at multiples of 3 and 4, 
with troughs at other values, especially at the higher end of the range), the research team used a 
predictive mean matching (PMM) method to impute the missing values. 


Interval-valued Variables 


Interval-valued variables are measures that are reported in intervals. The two such variables in this data 
are baseline and follow-up household income. At baseline, participants could only provide an interval 
response ($0; $1 to $9,999; $10,000 to $14,999; $15,000 to $19,999; $20,000 to $29,999; and $30,000 
or more). In the follow-up survey, respondents could either provide an exact amount or select one of the 
intervals. 


Interval regression was used to impute exact income from the interval responses, with the income levels 
specified in the log form. For the analysis of survey outcomes, the Stata MI IMPUTE INTREG command 
within the ICE specification was used to produce plausible values on the continuous scale of income 
within the specified range. Categories of income were then derived from the continuous values. To allow 
imputation of zero incomes, the log specification included an offset, so that the intermediate imputation 
variable had the form of log(income+offset) where offset was set to $8,000, which approximately 
corresponds to the 10th percentile of the imputed income distribution. 


Conditional Variables 


Some variables are only applicable to certain participants based on their responses to previous questions 
— for example, in the follow-up survey, questions about current employment were only asked of 
participants who were not still enrolled in their training program. Thus, the screener questions were 
imputed first, then the conditional variables were imputed for the participants who met the screener 
condition. 


B.3.4 TREATMENT OF THE HIERARCHICAL STRUCTURE OF THE DATA 


Standard errors and confidence intervals in Chapters 4, 5, and 6 reflect variability across programs. In 
order for the variability across programs not to be reduced by imputation, one option would be to only 
use information from other participants in the same program to impute missing data. However, the 
sample sizes are too small to make this a practical solution. As a compromise, the research team used 
program indicators as fixed effect predictors. This approach is known to inflate the variation between 
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groups (van Buuren 2018, sec. 7.3.2), which can be viewed as conservative in the context of this 
evaluation. 2 


Several small programs have about 15 participants, and about half a dozen programs with 25 to 
30 participants. Some of the smaller programs were combined with larger ones to avoid unnecessary 
losses in degrees of freedom: 


e Technology (Ivy Tech: Database Management, n=11; Server Administration, n=24; Informatics, 
n=25; Computer Science, n=26; Chaffey: Pre-Engineering, n=15) 


e Welding Technology (Manchester, n=21; and Washburn, n=27) 


e All others (Bossier: Fast Track to Manufacturing, n=19; Miami Dade: TRAMCON Basic, n=13; 
TRAMCON Advanced, n=10; South Central: Right Skills Now, n =10; Machining, n = 26; Chaffey: 
Mechanical Craft, n = 25) 


B.4. SURVEY NONRESPONSE ANALYSIS 


B.4.1 EVIDENCE OF NONRESPONSE BIAS IN UNADJUSTED OUTCOME MEANS 


A total of 2,767 adults filled out the BIF and 2,211 of them responded to the follow-up survey, yielding a 
response rate of 80 percent. To the extent that study participants who responded to the follow-up survey 
may be different from those who did not, there is a risk that estimates could be subject to nonresponse 
bias. 


We used demographic variables collected on the BIF to predict response to the follow-up survey. Missing 
information in the predictor variables were categorized into a separate category for purposes of 
modeling. The PROC GLMSELECT procedure in SAS was used to identify the most significant predictors of 
response to the follow-up survey. A series of models were run to identify potential predictors, including 
both Least Absolute Shrinkage and Selection Operator (LASSO) and forward stepwise methods. For both 
of the two methods, a model was run on the full sample of 2,767 baseline respondents and three 
additional models were run using the full sample randomly split into a training dataset and a validation 
dataset. The training datasets contained a random 80 percent of the sample and were used to train the 
models. The validation datasets contained the remaining 20 percent of the sample and were used to 
predict survey response. Exhibit B-5 compares the results of these eight models in terms of the 
significance of the baseline variables in predicting response to the follow-up survey. The starred predictor 
variables in the table were selected for use in constructing the nonresponse weights. 


2 An alternative approach, as described in Chapter 7 of van Buuren (2018), is to fit a multilevel model to each outcome 
(with individuals being level-1 units, and programs being level-2 units), generate random draws of parameter values and 
random effects, and generate imputed values as expected value plus noise (i.e., create posterior draws). While 
statistically appealing, this modeling approach is not computationally feasible for imputation of hundreds of outcomes. 
Moreover, some of the commonly used methods, such as predictive mean matching (PMM), arguably the most robust 
imputation method, do not have a natural multilevel extension. 
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Exhibit B-5. Significant Predictors of Response to Follow-up Survey by PROC GLMSELECT Model Type 
Lasso Method 


Predictor Variable 
from Baseline Survey 


Full 


Forward Stepwise Method 
80% Training / 20% Validation 


Full Sample 


80% Training / 20% Validation 


Sample 


Note: * denotes predictor was selected for use in constructing the nonresponse weights. 
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Industry of most recent Education, Education, Public Education, - 
employment* X X X X Public Admin/Admin, - Manufacturing/Mining, 
Admin/Admin Construction/Utilities | Public Admin/Admin 
eS Associate’s Degree, Associate’s Degree, vege 
Education Level* X X X ances ° Bachelor’s sea Bachelor’s pa pecueates 
pores, GED GED Master’s degree peice 
Home ownership X Own Live rent free 
Hispanic ethnicity 
Race X Asian 
Annual family income 
Reason for enrolling X X Career change Career change 
Marital status* X X X Living with Living with partner, Living with partner, ead 
partner, Married | Married Married : 
Married 
Military status No 
US citizen 
Non-English speaker at X 
home 
Sex* X X X 
Household received 
public assistance 
Grantee* X X X X Manchester Manchester Manchester 
Program HVAC 
Most recent hourly wage X X X 4+ times the 4+ times the 0-2 times, 3-4 times, 
(categorical) * minimum wage | minimum wage 4+ times min wage 
Expected Work Hours X X 
(categorical) 
Years of work experience 
in industry (categorical) 
Number of adults in 1 adult, 2 adults, 1 adult, 
household (categorical)* - . ‘ ‘ss 1 adult, 3 adults | 1 adult, 3 adults 3 adults 3 adults 
Nome! OF cilldrensin X X X X 3+ children 3+ children 3+ children 
household (categorical)* 
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B.4.2 CONSTRUCTION OF NONRESPONSE ADJUSTMENT WEIGHTS 


The final nonresponse adjustment weight was computed in two steps. First, a response propensity 
adjustment weight (WEIGHT1) was computed for the 2,767 BIF respondents. The final nonresponse 
weight (WEIGHT2) was computed for all 2,211 follow-up survey respondents which calibrates the 
response propensity weight of follow-up survey respondents to characteristics of the BIF respondents. 


The PROC LOGISTIC procedure in SAS was used to estimate a logistic regression model in which 
responding to the follow-up was regressed on eight characteristics measured in the BIF that were 
selected by the procedure described in the previous section: 


e Industry of most recent employment 

e Education level 

e Marital status 

e Sex 

e Grantee of the program attended 

e Hourly Wage of most recent employment (computed in comparison to minimum wage) 

e Number of adults in the household (1, 2, 3, 4+ adults) 

e Number of children in their household (0, 1, 2, 3+ children) 

The inverse of the raw estimated response propensities from this model were used to compute a 
response propensity weight (WEIGHT1) for all 2,767 BIF respondents, assigning a value of zero to 
nonrespondents. As a final step in the weighting, the characteristics of follow-up survey respondents 
were calibrated to match the distribution of the entire frame of BIF respondents. We used an iterative 


adjustment called raking? that aligned weighted totals of follow-up survey respondents with the baseline 
distribution on these seven dimensions: 


e Industry of most recent employment 

e Education level 

e Marital status 

e Sex 

e Grantee of the program attended 

e Number of adults in the household (1, 2, 3, 4+ adults) 

e Number of children in their household (0, 1, 2, 3+ children) 


The final raked nonresponse weight (WEIGHT2) is computed for all follow-up survey respondents (2,211) 
and scaled to sum to the total number of BIF respondents (2,767). Exhibit B-6 shows a summary of the 
nonresponse weights. 


3 A description of the raking procedure and SAS code is available here: https://www.abtassociates.com/raking-survey- 
data-aka-sample-balancing 
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Exhibit B-6. Description of Final Nonresponse Weights 


Min Max Mean 
Weight Weight Weight 
Value Value Value 
WEIGHT1 Response Propensity Weight 2,211 1.020 2.023 1.251 1.009 


WEIGHT2 Final Nonresponse Weight 2211 1.014 2.033 1.251 1.010 
(Follow-up respondents only) : 


Sample 
Size 


Approximate 
Design Effect 


Weight Variable 


B.5. INFERENCES ABOUT MEAN OUTCOMES FOR PROGRAMS 
LIKE THOSE FUNDED BY TAACCCT 


Parallel to the discussion in Section A.4 of Appendix A, Exhibit G.2 through G.5 of Appendix G contain 
confidence intervals for mean outcomes and Exhibit G.6 through G.9 of Appendix G contains confidence 
intervals for differences in mean outcomes across subgroups defined by participant characteristics at 
program entry. These were calculated using the same methodology discussed in Section A.4. They should 
help future program and grant designers communicate to future program participants and funders 
reasonable expectations for the mean outcomes of a new program or a new round of grant-funded 
programs. The confidence intervals are strictly valid only if the 34 studied programs can be viewed as a 
simple random sample of a broader universe of programs that could be created by the program designer 
or funded by future grant-funding mechanism. While this assumption is clearly not tenable, the 
confidence intervals should still lead to better decision-making than assuming either that the results of 
this study will be exactly replicated or that nothing at all is known about the likely outcomes. Also, clearly, 
the confidence intervals will be more useful if the future program is implemented under the same general 
economic conditions that prevailed in the late 2010s, i.e., late in a multi-year economic expansion. 


These confidence intervals were calculated with the aid of two SAS procedures, MIANALYZE for multiple 
imputation analysis, which in turn called SURVEYMEANS to produce results for each imputation (a set of 
completed data). The calculations involved corrections for the extra uncertainty introduced by clustering, 
weighting and imputation. The 34 programs were treated as clusters. Subgroups were treated as 
domains. The formulas for multiple imputation in Section A.4 also apply here. 
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C. Unemployment Insurance Wage 
Data 


This appendix describes the Unemployment Insurance (UI) wage data collected for study participants 
from the National Directory of New Hires (NDNH), a centralized database operated by the federal 
Department of Health and Human Services Office of Child Support Enforcement (OCSE). The NDNH 
contains quarterly wage information submitted by state workforce agencies. OCSE also supplements the 
state reports with records about earnings from federal civilian and military jobs (which are otherwise not 
covered by state UI data). Section C.1 describes the data collection process. Section C.2 provides detail 
on the construction of the measures. 


C.1. DATA COLLECTION PROCESS 


The primary purpose of the NDNH is to assist state child support agencies to locate a non-custodial 
parent living or working in a different state in order to establish or enforce a child support order. 
However, subject to federal law and other requirements to protect data privacy and security, OCSE may 
disclose certain information contained in the NDNH to local, state, or federal agencies for research 
purposes. 


DOL and OCSE negotiated a memorandum of understanding (MOU) allowing access to NDNH data for the 
TAACCCT evaluation. Among other provisions, the MOU specifies the participant-level data that may be 
merged with NDNH data and procedures for maintaining the security and confidentiality of the data. 


There is a two-step process for collecting NDNH data for research. First, the research team transmitted 
“match” request files to OCSE. These match request files contain the names and SSNs of study 
participants. OCSE verifies with the Social Security Administration that the reported SSNs belong to the 
named persons. For those SSNs that are successfully verified, OCSE copies all earnings records from the 
NDNH and makes them available for DOL to save in a secure folder. These copied records contain a 
pseudo-SSN; the records are stripped of all personal identifiers. 


The NDNH database only maintains the most recent two years of earnings data, so each match file 
returns about eight quarters of data. In order to be able to analyze the longest period of data possible, 
the research team submitted match request files quarterly, beginning in late 2016, when participants first 
enrolled in the study, and continuing through September 2019. 


The second step of the data collection process involves merging study data on to the wage files. The 
MOU between DOL and OCSE identifies a specific set of baseline and follow-up survey variables that can 
be merged with the wage files, after OCSE verifies that the variables are formatted according to the 
approved layout. The research team submitted this “passthrough” file to OCSE. OCSE then strips the 
personal identifiers out of the passthrough file and replaces the actual SSN with the same pseudo-SSN 
previously assigned to the archived wage records. The research team then used this pseudo-SSN to 
merge baseline and follow-up survey data with the quarterly wage data in order to conduct analysis on 
DOL's secure server. 
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NDNH earnings data was only available for participants who provided valid names and SSNs in the BIF 
that could be validated by the Social Security Administration. Like other items on the BIF, SSN was an 
optional item for participants, and a sizeable number did not provide an SSN. As a result, NDNH earnings 
data was only available for 2,355 participants who provided a verified name and SSN, or about 

85 percent of the full study sample of 2,767. 


C.2. DETAILS ON MEASURES 


Enrollment into the study began in August 2016 and ended in October 2017. Given the lag of up to six 
months in processing of employer reports by the states and transfer of state data to OCSE, wage records 
from NDNH were available through Quarter 1 2019 (March 31, 2019). This provided five quarters of post- 
enrollment data for everyone in the sample with a verifiable name and SSN, and up to 10 post-enrollment 
quarters for those enrolled earlier in the period. This report also includes earnings in the quarter of 
enrollment and the first three pre-enrollment quarters. 


Note that pre-enrollment and post-enrollment quarters are defined relative to the calendar quarter of 
program entry. For example, participants who enrolled between January 1, 2017 and March 31, 2017 
were enrolled in Quarter 1 2017, which is designated as their quarter of program entry. For these 
participants, their fifth post-enrollment quarter is Quarter 2 2018. 


Quarterly earnings were calculated by adding up earnings reported across all jobs held in that quarter. 
Participants were considered to be employed if they had any earnings during the quarter, and not 
employed if they had zero earnings during the quarter. Exhibit C-1 contains a description of the NDNH- 
based earnings and employment measures. 


Exhibit C-1. Description of NDNH-based Earnings and Employment Measures 


Measure Definition Source 
Total earnings from all jobs reported during a calendar quarter. 
Reported for the following quarters: 

e 3rd quarter before program entry 

e 2nd quarter before program entry 


1st quarter before program entry 

Quarter of program entry National 
Quarterly earnings ist quarter after program entry Directory of 

2nd quarter after program entry New Hires 


3rd quarter after program entry 
4th quarter after program entry 
5th quarter after program entry 


Missing for individuals without a valid name and SSN. 
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Measure Definition Source 
Binary indicator for any earnings during a calendar quarter. 
Reported for the following quarters: 

e 3rd quarter before program entry 


3rd quarter after program entry 
4th quarter after program entry 
5th quarter after program entry 


e 2nd quarter before program entry 

e 1st quarter before program entry 

e Quarter of program entry National 
Quarterly employment e 1st quarter after program entry Directory of 

e 2nd quarter after program entry New Hires 


Missing for individuals without a valid name and SSN. 


Change in quarterly earnings 
between the 3rd quarter National 
before program entry to the Directory of 


5th quarter after program a eee ; : New Hi 
ae a Missing for individuals without a valid name and SSN. as 


Total earnings in the 5th quarter after program entry minus 
total earnings in the 3rd quarter prior to program entry. 


C.3. IMPUTATION OF MISSING DATA 


All analyses of quarterly earnings and employment outcomes used the set of 2,355 participants with 
complete NDNH data. The research team did not impute any of the NDNH variables, but did impute 
missing baseline and follow-up survey measures that were used in analysis. They imputed these data in 
three distinct operations on different platforms: Once on the Abt Associates secure server (ACE3) for 
Research Questions 4, 5 and 6; once on the DOL SAS server for Research Question 7; and once on a DOL 
container for RStan for Research Question 8. If the ACE3 imputation had been conducted before creating 
the passthrough file, this three-fold imputation could have been avoided. However the schedule did not 
permit this approach. Moreover, the three-fold imputation allowed better customization of the imputation 
procedures to the research questions. The research team used SAS PROC MI to impute missing baseline 
subgroups and service receipt outcomes for the analysis in Chapter 6, using the fully conditional 
specification (FCS) approach described in Appendix B.3. R statistical software was used for imputation of 
baseline data for the analysis of outcomes by program in Chapter 7, using the Multivariate Imputation by 
Chained Equations (MICE) package (van Buuren 2018). This imputation used logs of NDNH earnings in 
Quarters —3, 0 and +5 as variables in the model; Quarter 0 is the natural baseline; Quarter 5 is the 
natural outcome; and Quarter —3 is used to make the imputation model consistent with the outcome 
eventually modeled in small area estimation for Chapter 7 (growth in earnings between Quarter —3 and 
Quarter 5). 
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D. Service Impacts 


This appendix describes procedures used to estimate the impacts of service receipt on outcomes reported 
in Chapter 6. Section D.1 describes the general methodological approach. Section D.2 provides specific 
model results for each outcome. Section D.3 discuss the assumptions required for the impacts to be 
interpreted as causal and potential reasons why those assumptions may not necessarily hold. 


D.1. GENERAL METHODOLOGY 


The procedure used to estimate the impact of service receipt on outcomes involves calculating 
regression-adjusted differences between those who received and those who did not receive each 
particular service. In an attempt to make the process more transparent to readers not acquainted with 
logistic or linear regression, the team fit these models in a novel manner involving four steps and visual 
aids. The text below includes a brief summary of the steps, followed by more detail about each. 


D.1.1 BRIEF SUMMARY OF STEPS 
1. Determine which exogenous factors are relevant to the outcome; 


2. Fit a working model for the outcome in terms of only the relevant exogenous factors while ignoring 
service receipt patterns; 


3. Use the working model to calculate the predicted outcome level, separately for those who received 
each service and those who did not; 


4. Calculate the difference between the actual and predicted outcome levels, separately for those who 
received each service and those who did not. The difference in these differences is the estimated 
impact of receiving the particular service. 


D.1.2 DETAILS ON STEP 1 


The team started from a large collection of potentially relevant exogenous factors: 


e Age (4 categories) 

e Gender 

e Race/ethnicity (4 categories) 

e Family structure (4 categories) 

e Educational attainment at program entry (6 categories) 

e Family income at program entry (both a 3-level categorical variable and a continuous variables) 
e Housing tenure (3 categories) 

e Receipt of Trade Readjustment Allowances at program entry 

e Receipt of Supplemental Nutrition Assistance Program (SNAP) benefits at program entry 

e Receipt of Temporary Assistance for Needy Families (TANF) benefits at program entry 


e Employment status/length of unemployment at program entry (4 categories) 
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e Primary reason for program enrollment (6 categories) 

e Expected work hours during training (4 categories) 

e Citizenship 

e Veteran status 

e Language spoken at home (2 categories, English or non-English) 

e Plan for full-time study 

e Years of experience in target industry (3 categories) 

e Current/last industry (10 categories) 

e Family poverty status at program entry 

e A binary flag for whether target industry is goods producing (versus services) 

e Expected duration of program in months 

e County employment rate among the working-age population* 

Out of a desire to avoid problems of multicollinearity and to simplify the presentation of prediction 
models, the team first winnowed this long list down to a few critical exogenous factors for each outcome. 


The team did this separately for each of five outcomes—program completion, training-related 
employment, change in earnings, public assistance benefit receipt, and poverty. 


For this winnowing process, the team used relatively recently developed technique to determine which 
exogenous factors are relevant to the outcome. The technique is Known as “least absolute shrinkage and 
selection operator” (LASSO) with “10-fold cross-validation.”> With the LASSO, the sum of absolute values 
of the estimated regression coefficients in a proposed model is constrained to be less than a preselected 
value (the “constraint”). If the value for this constraint is small enough, many coefficients in the proposed 
model will be forced to zero in order to fit within the cap on the sum of absolute coefficient values and 
thus can be removed from the list of baseline covariates. The 10-fold cross-validation is used to optimize 
the value of the constraint, rather than just relying on an arbitrary choice for it. 


Details of the procedure are as follows: 


1. With 10-fold cross-validation, the sample is divided into 10 equal and mutually exclusive random 
subsamples. 


2. For each of a range of candidate values of the constraint, the LASSO procedure is run to select 
covariates on a sample in which one of the 10 subsamples has been dropped. 


3. A linear model is fit on the same sample using just the variables selected in the second step for each 
of the candidate values of the constraint. 


4 This variable was not available for the earnings analysis. 
5 See Buhlmann and van de Geer (2011) for a full explanation of these techniques. 
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4. The model is used to create out-of-sample predictions of the outcome for everyone in the dropped 
piece of the sample, and the prediction error Y“_i-Y_i is measured for each of the candidate values of 
the constraint. 


5. Steps 2 through 4 are repeated 10 times for each candidate value of the constraint. On each 
iteration, a different one of the 10 subsamples is dropped. In this manner, out-of-sample prediction 
errors are obtained for the entire sample. 


6. Mean squared prediction errors across all 10 replicates are then calculated for each of the candidate 
values of the constraint. 


7. The value of the constraint that minimizes this cross-validated mean squared prediction error and 
thus captures most of the variation reduction possible with the available covariates is selected as the 
optimal constraint.© Whichever variables have nonzero coefficients in the model for that optimal 
constraint are selected as relevant to the outcome. All other exogenous factors are ignored. All of this 
is done automatically in SAS®/GLMSELECT. 


A couple of technical notes about GLMSELECT. First, it does not support a logistic LASSO, only a linear 
logistic. This is not a serious problem, but it does mean that some of the selected variables turn out not 
be to statistically significant in the final logistic regression model. Second, the cross-validated LASSO 
option in GLMSELECT works on dummy variables created for each level of multi-level categorical 
variables. It ignores the larger framework surrounding each dummy. This means that, for example, it 
may select just one of the 10 current/prior industries or just one race/ethnicity level for inclusion in the 
set of relevant exogenous variables. 


Finally, it is worth noting that team repeated the LASSO separately for each of five multiply-imputed 
datasets. This should capture the extra uncertainty due to winnowing variables based on their imputed 
values. 


D.1.3. DETAILS ON STEP 2 


For binary outcomes, the team fit the logistic model in equation C-1. The team did this separately for 
each of five multiple imputations. 


eel eae (C-1) 


where 


/indexes program, 


Y, is a binary outcome indicating whether the person experienced the outcome (completing the program, 


obtaining training-related employment, having income below the poverty threshold, or receiving public 
assistance benefits, 


6 One could simply use the LASSO to select covariates with a pre-specified value of the constraint, but the 10-fold cross- 
validation provides a principled method for selecting the constraint. 
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Pry Y, =1 
A, = log et is the logit probability of the person /experiencing the outcome, 


i077?» X, 1S a collection of p baseline covariates such as race and dummy variables for each program, 


pi 


A, fis, ate unknown coefficients to be estimated. 


For change in earnings, the team fit the linear model in equation C-2. The team also did this separately 
for each of five multiple imputations. 


YsutXy nytt Xi, TG, (C2) 
where 


Y, is the change in earnings over the two-year period from three quarters prior to program entry to the 


fifth quarter following program entry and 
é, is a normally distributed random error. 


The team fit models C-1 and C-2 on the total sample, ignoring receipt of services. Therefore, the 
estimated regression coefficients reflect that average contribution of exogenous factors across observed 
(but ignored) service-receipt conditions. 


D.1.4 DETAILS ON STEP 3 


For the binary outcomes, once equation C-1 had been fit, the predicted probability of a person 
experiencing the outcome was calculated as: 


1< 42 
Ut) f 


r=1 


where 


g, indicates whether the person received service g, and 


W, is a nonresponse-adjustment weight. 


Similarly, for change in earnings, the “excess increase in earnings” was calculated as 
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The impact of receiving the service was then calculated as 
I = B,- B, 


This same equation was used for both binary outcomes and change in earnings. 


The variance on these estimated impacts was calculated using a combination of SAS/SURVEYMEANS and 
SAS/MIANALYZE. 


With SURVEYMEANS, the team estimated the “full-sample” variance for a single one of the multiple 
imputations. It was run with cluster=program so as to reflect the extra uncertainty caused by the 
clustered nature of the sample. The nonresponse weights were also used. Let this estimated variance be 
With 


denoted as and let the corresponding impact (based on that imputation) be / 


g(r) g(r)" 


MIANALYZE, the cross-imputation variance was calculated as: 


Lie —\2 
=) iets) 


r=1 
where I, is the average estimated impact across the five multiple imputations. 


Finally, the total variance of the estimated impact is: 
Vi= ly or 
gg 5 2 Qeen +S g 


Regarding transparency with a non-technical audience, note that the estimated impact (for either a 
binary or continuous outcome) should be similar to what would be obtained from a linear model of the 
form C-3. 


YeutXyy, t+ X74, +1,8; 76, (C-3) 


Our hope though is that displaying expected mean outcomes, actual mean outcomes, their differences, 
and the difference of the difference will give the non-technical reader better insight into the nature of the 
causal machinery. 


D.2. MODEL RESULTS BY OUTCOME 


This section displays the working models used for each outcome. There was a slightly different model fit 
for each multiply imputed dataset. This section displays models fit on the first multiple imputation of the 
data. 


Exhibit D-1 shows the model fit for program completion. The model finds that several exogenous 
characteristics are associated with lower rates of program completion, including: younger participants age 
20 or less (-0.26); those who expect to work at least 20 hours per week in the next few months (-0.22); 
those with current or prior employment in the accommodation or food service industry (-0.35); and 
expected program duration in months (-0.19). Characteristics associated with higher rates of program 


Abt Associates Round 4 Early Outcomes Study Report December 2020 1 D-5 
Technical Appendices 


APPENDIX D 


completion include sex (female, 0.43); prior college credit (0.25); not currently employed but worked in 
the prior year (0.19); prior employment in the healthcare or social assistance industry (0.71); and 
enrollment in a program targeting a goods-producing industry. 


These results generally appear to be plausible and in line with expectations. For example, participants 
who plan to work at least 20 hours a week during training would be expected to have less time to spend 
on their classes and have lower completion rates than those who do not plan to work. In addition, 
participants with prior college credit have shown they can be successful in a college setting and thus are 
more likely to complete their programs. 


Exhibit D-1. Model for Program Completion in Terms of Exogenous Factors 


Regression Standard 


Exogenous Factor Coefficient error p-Value 
Intercept 0.26 0.65 .690 
Age 20 or less -0.27 0.12 .020 
Female 0.43 0.15 .003 
Highest level of education at program entry: 

Technical, trade or vocational degree 0.35 0.22 .103 

Some college credit, but no degree 0.25 0.11 .020 
Employment history at program entry: 

Not currently employed but employed in last year 0.19 0.10 .066 
Expecting to work at least 20 hours per week in the next few months -0.22 0.12 .063 
alin or recently employed in accommodation or food service 0.35 0.16 031 
hace or recently employed in healthcare or social assistance 0.71 0.22 001 
Target industry is goods producing 0.68 0.10 <.001 
Expected program duration in months -0.19 0.01 <.001 
County employment ratio for working age population 0.94 0.93 313 


Exhibit D-2 shows the model fit for training-related employment. Several characteristics are associated 
with lower rates of training-related employment, including race (non-Hispanic black, -0.69); annual family 
income less than $15,000 (-0.34); SNAP receipt (-0.54); no experience in target industry (-0.67); and a 
transformed measure of expected program duration in months (-0.30). Characteristics associated with 
higher rates of training-related employment include speaking a language other than English at home 
(0.43); planning to study full-time (0.47); career advancement as the most important reason for enrolling 
in training (0.22); prior employment in the healthcare or social assistance industry (0.85); enrollment in a 
program targeting a goods-producing industry (0.47); and those age 25 to 34 who were employed at 
program entry (0.40). 


These results generally seem to be plausible — economically disadvantaged groups and those with no 
experience in the target industry have lower rates of training-related employment, while those who plan 
to study full-time, were already employed at program entry, and who rated career advancement as the 
most important reason for training had higher rates of training-related employment. 
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Exhibit D-2. Model for Training-Related Employment in Terms of Exogenous Factors 


Regression Standard 


Exogenous Factor Coefficient pe p-Value 
Intercept -2.20 0.78 .005 
Black (Non-Hispanic) -0.69 0.15 <.001 
Annual family income less than $15,000 -0.34 0.13 .008 
SNAP receipt at program entry -0.54 0.19 .005 
Speaks language other than English at home 0.43 0.12 <.001 
Plans to study full-time 0.47 0.11 <.001 
Most important enrollment reason was career advancement 0.22 0.12 .060 
No experience in target industry -0.67 0.11 <.001 
sar el or recently employed in healthcare or social assistance 0.85 0.19 <.001 
Target industry is goods producing 0.47 0.11 <.001 
erie program duration in months (squared and then divided -0.30 0.07 <.001 
Aged 25 to 34 and currently employed at program entry 0.40 0.14 .004 
County employment ratio for working age population 2.11 1.08 502 


Exhibit D-3 shows the model fit for change in earnings. The regression coefficient gives the effect of the 
exogenous factor on the change in earnings; a positive coefficient indicates a larger change in earnings, 
while a negative coefficient indicates a smaller change in earnings. 


Exhibit D-3. Model for Change in Earnings in Terms of Exogenous Factors 


Regression Standard 


Exogenous Factor Coefficient Error p-Value 
Intercept $3,916 $307 <.001 
Not currently employed but employed in last year -$1,422 $304 <.001 
Not expecting to work for pay in coming months at program entry -$1,357 $349 <.001 
Expected program duration in months $135 $30 <.001 
Quarterly wages for 3 quarter prior to program entry (in thousands) -$364 $46 <.001 


Notes: A positive regression coefficient indicates that the exogenous factor is associated with a larger change in 
earnings, while a negative coefficient indicates that the exogenous factor is associated with a smaller change in 
earnings. 

The coefficient in the last row indicates that each additional $1,000 in earnings in the 3rd quarter before program 
entry is associated with a $364 smaller increase in earnings 


Since this outcome measures the difference in earnings between the fifth quarter after program entry 
and the third quarter before program entry, the exogenous factors selected by the model may influence 
the outcome by their effect on either pre-enrollment or post-enrollment earnings. Two of these measures 
appear to affect the outcome through their influence on pre-enrollment earnings: those not currently 
employed but who worked in the last year had a much smaller increase in earnings ($1,422 smaller) than 
other groups, likely because participants in this group had high pre-enrollment earnings. More directly, 
those with higher quarterly wages in the third quarter before program entry had a smaller increase in 
earnings ($364 smaller increase for each additional $1,000 in the third quarter before program entry). 
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Two other factors are associated with the change in earnings. Those not expecting to work for pay in the 
coming months at program entry had a smaller increase in earnings ($1,357 smaller), perhaps reflecting 
a weak connection to the labor market. Expected program duration in months is associated with a larger 
increase in earnings ($135), which could be due to greater labor market returns from longer training 
programs. 


Exhibit D-4 shows the model fit for receipt of poverty. A number of demographic and socio-economic 
characteristics are associated with higher rates of poverty at follow-up, including: race/ethnicity (non- 
Hispanic black), living with own children but no spouse or partner; low levels of education; low levels of 
family income; receipt of public assistance benefits; and expectations for no or limited work hours in the 
coming months. Characteristics associated with lower rates of poverty include: non-Hispanic white; living 
with spouse or partner but no children; owning a home; being employed at program entry; and family 
income. 


Exhibit D-4. Model for Poverty in Terms of Exogenous Factors 


Regression Standard 


Exogenous Factor Coefficient Error p-Value 

Intercept -0.26 0.32 424 
Race/ethnicity: 

Black (Non-Hispanic) 0.29 0.14 .038 

White (Non-Hispanic) -0.54 0.12 <.001 
Family structure: 

Lives with own children but no spouse or partner 0.40 0.15 .007 

Lives with spouse or partner but no own children -0.37 0.18 .039 
Did not graduate high school and no GED 0.58 0.25 .023 
Annual family income less than $15,000 0.89 0.30 .003 
Annual family income between $15,000 and $29,999 0.42 0.22 .057 
Homeowner -0.32 0.16 .044 
SNAP receipt at program entry 0.60 0.16 <.001 
Employed at program entry -0.38 0.12 .001 
Not expecting to work for pay in coming months at program entry 0.32 0.13 .018 
Expecting to work at 1-19 hours per week in the next few months 0.68 0.21 .001 
Family income (in ten thousands) -0.20 0.06 <.001 
Living in poverty and program entry 0.07 0.16 .672 


Exhibit D-5 shows the model fit for receipt of public assistance benefits. A number of socio-economic 
characteristics have a relationship with receipt of public assistance benefits. Participants with more 
education; who were employed at program entry, and who had higher levels of family income have lower 
rates of public assistance benefits receipt. Participants who had lower levels of family income; were 
receiving TRA, SNAP, or TANF at program entry; who did not expect to be working for pay in the coming 
months; and who previously served in the military had higher rates of public assistance benefit receipt. 


Participants enrolled in longer programs tended to have slightly higher rates of benefits receipt—perhaps 
due to loss of income from being enrolled in a longer program. Living rent-free at program entry is 
associated with a lower rate of public assistance benefit receipt; perhaps these were younger participants 
living with their parents at program entry. 
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Exhibit D-5. Model for Receipt of Public Assistance Benefits in Terms of Exogenous Factors 


Regression Standard 


Exogenous Factor Coefficient enor p-Value 
Intercept -1.01 0.86 .240 
Highest degree at program entry was AA -0.75 0.28 .008 
Annual family income less than $15,000 0.21 0.17 .198 
Live rent-free at program entry -0.43 0.15 .004 
TRA receipt at program entry 1.69 0.38 <.001 
SNAP receipt at program entry 2.16 0.17 <.001 
TANF receipt at program entry 0.68 0.50 177 
Employed at program entry -0.31 0.14 .026 
Not expecting to work for pay in coming months at program entry 0.37 0.15 .013 
Veteran 0.89 0.20 <.001 
Family income (in ten thousands) -0.10 0.03 <.001 
Expected program duration in months 0.03 0.01 .001 
County employment ratio for working age population -1.03 1.22 399 


D.3. ASSUMPTIONS REQUIRED FOR CAUSAL INFERENCE 


The methods described in this appendix are asymptotically equivalent to using equation C-3 for inference. 
As is well known, estimates of / z for equation C-3 can be consistent estimates of the average treatment 


effect under the conditional independence assumption (CIA) (Angrist and Pischke, 2008). Under this 
assumption, the potential outcome under service receipt must be conditionally independent of service 
receipt given the covariates in the model, and the potential outcome under lack of service receipt must 
also be conditionally independent of service receipt given the covariates in the model. The team 
anticipated doing this kind of analysis when designing the BIF and endeavored to capture the variables 
that would make the CIA at least somewhat plausible. However, as discussed in the main body of the 
text, there are several plausible phenomena that would lead to violations of CIA in this application. The 
text discusses these possibilities in more detail—particularly the box Analytic Methods and 
Interpreting Service Receipt Impacts in the introduction of Chapter 6. 


Some researchers may wonder why we did not use some other methodology. There are many available 
choices. One popular choice is propensity matching (also explained in Angrist and Pischke, 2008, among 
many other papers and textbooks). This method involves modeling service receipt rather than the 
outcome. The team decided against propensity matching because we had a larger number of services of 
interest than outcomes. By modeling four outcomes rather than 11 services and four service bundles, we 
reduced the modeling effort. Both methods are equally vulnerable to violations of the CIA. 
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E. Methodology for Estimating 
Outcomes by Program 


This Appendix describes statistical methodology used to address Research Question 8: 


How do success rates (program completion, employment in targeted field and earnings) 

vary across programs and grantees? 
This Appendix is organized as follows. Section E.1 sketches the methodology. Section E.2 illustrates how 
the complex methodology improves upon naive estimates of local outcomes. Section E.3 goes into full 
detail on the methodology. This last section speaks to a more specialized audience than the earlier 
sections. It contains details sufficient for understanding and reproduction of our work by other 
statisticians. Finally, Section E.4 provides a technical discussion of the application of the methodology to 
TAACCCT. Most critically, this section discusses the evidence about the true level of outcome variation 
across TAACCCT programs. 


E.1. METHODOLOGY SKETCH 


One approach to estimating local outcomes would be to simply report the observed mean for each 
program. The team did not use this approach in Chapter 7 of this report because many of the programs 
had very small sample sizes. As such, the simple reported means would be very noisy and therefore poor 
predictors of how future cohorts of students might fare at these programs. As discussed in Chapter 5, 
outcomes varied considerably by baseline characteristic. Assuming that these relationships are fairly 
stable across programs and that the profile of students attracted to a program in the future, the team 
used these relationship to adjust the local means. These profile variables are shown in Box E-1. 
Moreover, the team used estimates of the measurement error on each naive local mean to estimate the 
likely true long-run future variation in mean outcomes across the programs. The team used this 
estimated true cross-program variation to further adjust the local means so that the adjusted estimate for 
each program is the “best” predictor of the true long-run outcome for that program. The word “best” 
here has a specific meaning in statistical theory as is explained at greater length in Section E.3, but it 
loosely means that statisticians do not know how to prepare a better prediction for the program given the 
available information and a particular framework for judging quality. 


As it turns out, statisticians are divided into three competing schools of how to prepare these “best” 
estimates based on competing understandings of what is meant by the concept of probability and how 
best to incorporate theories about the underlying processes as well as the design of data collection. The 
team used an approach from what is known as the Bayesian school of statisticians, named after a 

18‘ century cleric and philosopher by the name of Reverend Thomas Bayes. In this school of thought, 
the probability that a participant will, for example, complete a particular program is not a fixed unknown 
number. Rather, this probability is a random quantity that has uncertainty associated with it. Different 
researchers can have different prior sets of beliefs about this probability before the data are observed. 
These prior beliefs are expressed in the form of a prior distribution. After data are observed, these beliefs 
are updated following certain mathematical rules spelled out by Bayes theorem. These updated beliefs 
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are expressed in the form of a posterior 
distribution. The mean, variance and 
percentiles of this posterior distribution are 
commonly reported. 


None of the other analyses in this report 
use a Bayesian approach. Instead, the 
team used frequentist approaches for the 
other chapters. Chapters 4 and 5 use the 
“design-based” frequentist approach and 
Chapter 6 uses a “model-based” 
frequentist approach. These are the other 
two competing schools of statistical 
thought. In both frequentist schools, the 
probability of a particular participant 
completing a particular program is a fixed 
but unknown constant. In the design- 
based frequentist approach, the analyst 
makes no assumptions about underlying 
processes other than that there are no wild 
true outliers in the population like program 
participant getting a 7-figure annual salary 
as a result of completing one of the 
studied programs. In the model-based 
frequentist approach, the analyst makes 
assumptions about the process leading to 
the fixed but unknown parameters of 
interest. Examples of these assumptions 
include things like assuming that change in 
earnings is normally distributed (i.e., 
follows a bell-shaped curve) or assuming 
that the effect of a service on the 
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Box E-1. 
Student Profile Variables 


Age: (4 levels) 

Female 

Race/Ethnicity: (4 levels) 

Family structure: (4 types) 
Educational Attainment: (6 levels) 
Family Income: (3 levels) 

Home tenure: (3 levels) 

Public assistance: (four programs) 


Employment status & duration of unemployment: 
(4 levels) 


Expect Work Hours while in training: (4 levels) 
Citizenship & Veteran status: (3 levels) 
Language Other than English spoken at home 
Plans to study Full Time 

Primary Reason to Enroll: (5 reasons) 

Work Experience: (3 levels) 

Poverty status at Baseline: in poverty 


Training Targets the Goods-producing Industry 
Super Sector 


Program Length, Months 


Current or Previous Employment in Manufacturing 


Additional Student Profile Variables available for 
likelihood of program completion does not outcomes other than earnings 


vary by program. 


Cis Moyel-Tainalsa mem relaali \miacee)na(c 


e Local employment rate (county-level employment 
as percent of total population 18-64) 


In most of their professional work, the 
team prefers the design-based frequentist 
set of rules and customs for analysis. 
However, the work in Chapters 6 is 
focused on understanding underlying processes, so it made sense to use model-based frequentist rules 
and customs for it. All three schools of statistical thought can produce estimates of mean outcomes by 
program. The team found the use of the design-based frequentist approach to be inadequate for this task 
because this method only uses local information. As a result, if the sample size is small (as it is for many 
programs studied in this report), the estimate will have such a high variance as to be practically useless. 
Model-based frequentist methods can fix this issue, but the team rejected them for the analyses in 
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Chapter 7 because of the methodological problems in the available variance estimates on local outcomes. 
These methods are forced to neglect that the uncertainty caused in adjusted local means by having to 
estimate the true between-program variance. Only Bayesian methods can both reduce the variances 
enough to make the local estimates useful and also provide fair variances on those adjusted local 
estimates. 


Section E.2 illustrates the properties of the Bayesian estimates of local outcomes in contrast to those of 
the design-based frequentist approach, labeled as “direct” estimates. The direct estimates fully captures 
the idiosyncratic features of each program but will generally have large variances, as the sample sizes of 
programs are not particularly large. The Bayesian methods combine idiosyncratic results with what we 
should expect based on the profile of program participants and statistical models for how the 
characteristics of participants at program entry influenced their outcomes. These models were 
generalizations of simple regression models. In addition to student profiles, the models also used 
program duration and the local employment rate in the working age population.” 


Generally speaking, we expect that Bayesian estimates will improve upon the direct estimates in two 
ways. First, the Bayesian estimates will be better aligned with expected results based on the participant 
profile, program duration, and the local employment rate. Second, the width of the error bands will be 
much shorter. When the statistical model has little to no explanatory power (i.e., outcomes cannot be 
predicted by demographics and baseline variables), Bayesian estimates will be very similar to the design- 
based estimates, with some shrinkage towards the overall mean. When the model is highly predictive, 
Bayesian intervals will be shorter, especially for programs that have low sample sizes, but have the 
participant profile similar to that in some other programs in the study. 


It is important to note that the Bayesian estimates can also be worse than direct estimates. This can 
happen two different ways. First, it could happen that one program does a much better or worse job of 
serving a particular participant group (such as men with education below high school) than is the case at 
all or most of the other programs. If this happens, the adjustment is based on a faulty assumption and as 
a result, the adjusted estimate may be further from the truth than the direct estimate. Second, it can 
happen that the true distribution of participant means across programs is different than the assumed 
distribution. For example, instead of the means being normally distributed across programs with constant 
variance, it might happen that there are two groups of programs, one that turns out certified nursing 
assistants who make $11 per hour and one that turns out bachelors of chemical engineering that made 
$60 per hour. If this bifurcation was not part of the prediction model, then the mean participant-level 
residual at the program level would not follow a bell-shaped distribution. Instead, there would be two 
bells, sort of like a Bactrian camel. Assuming one hump when there are really two would result in the 
Bayesian procedure shrinking the program estimates for both groups of programs into the valley between 
the two humps, probably resulting in all of them being worse than the direct estimates. While the team 
checked the distribution of program-level mean person-level residuals for signs of non-normality, 
departures from normality are difficult to detect with just 34 programs. 


7 Technical constraints preventing merging geographic information on local employment rates to the earnings data in the 
NDNH, but this variable was used for the other outcomes. 


Abt Associates Round 4 Early Outcomes Study Report December 2020 1 E-3 
Technical Appendices 


APPENDIX E 


E.2. RESULTS 


We present the key results graphically in the form of comparison of direct and Bayesian estimates. We 
use the same style plots as used in Chapter 7, but overlay the two competing estimates. These plots are 
known as caterpillar plots for obvious reasons. They involve sorting the programs from worst to best— 
based on the Bayesian estimates of the local program mean. Note that caterpillar plots in Chapter 7 
showed 80 and 99.9 percent credible intervals. The plots below directly display the 95% confidence 
intervals computed from the design-based estimates and standard errors, which only use information 
contained solely within a given program; and model-assisted 95% Bayesian credible intervals that 
incorporate a model for the outcome. To the extent that the model is predictive of the outcome, we hope 
to see that the latter intervals are shorter. The design-based frequentist interval reported is the Wilson 
interval (Dean and Pagano 2015). This section shows these overlaid caterpillar plots for the same four 
outcomes studied in Chapter 7. In each of these, the orange squares mark the Bayesian estimates, the 
orange lines mark the Bayesian 95 percent credible intervals, the blue dots mark the direct estimates, 
and the blue lines mark the direct 95 percent confidence intervals. 


E.2.1 PROGRAM COMPLETION 


Exhibit E-1 shows the two sets of estimates of program completion rates by program. For this outcome, 
the two sets of estimates are fairly similar. Few programs would be ranked differently using the direct 
estimates and the credible intervals are not much shorter than the confidence intervals. For example, the 
short-term CDL and Forklift programs at Cincinnati State have the highest completion rates under either 
estimation strategy, and the two-year Server Administration program has the lowest. The reason that two 
sets of estimates are similar for this outcome is that between-program variance was estimated to be very 
large, indicating that it was dangerous to use cross-program information to adjust the estimates very 
much. 


Exceptions to this general rule exist. For example, the Bayesian procedure estimates a lower completion 
rate for the Welding Technology program at Manchester Community College than estimated by the 
design-based frequentist procedure. In the other direction, the Bayesian procedure estimates a higher 
completion rate for Right Skills Now program at South Central College than estimated by the design- 
based frequentist procedure. Another exception concerns the width of the intervals for Database 
Management program at Ivy Tech. The design-based frequentist procedure is blind to completion rates at 
other programs and thinks that the completion rate at this program could be anywhere between 0 and 

30 percent. In contrast, the Bayesian procedure looks at completion rates at other programs and 
concludes that the completion rate is unlikely to be larger than 15 percent. 
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Exhibit E-1. Program Completion Rate, by Program - Overlaid Bayesian and Direct Estimates 


CDL and Forklift (Cincinnati) - 

Core Plus (Delgado) - 

Licensed Practical Nurse (Washburn) - 
Mechanical Craft (Chaffey) - 

Emergency Medical Technician (Washburn) - 
Welding Technology (Washburn) - 

Advanced Manufacturing and Engineering (Chaffey) - 
Advanced Manufacturing (Manchester) - 
Welding Technology (Manchester) - 
Industrial Automation (Chaffey) - 

Industrial Maintenance (Chaffey) - 

TECH 101 (Delgado) - 

TRAMCON Advanced (Miami Dade) - 
TRAMCON Basic (Miami Dade) - 
Pre-engineering (Chaffey) - 

TRAMCON Foundation (Miami Dade) - 
Welding (South Central) - 

Advanced Welding Bootcamp and Program (Bossier) - 
Fast Track to Manufacturing (Bossier) - 
Certified Production Technician (South Central) - 
Right Skills Now (South Central) - 

HVAC (Chaffey) - 

Gateway Courses to IT Programs (Ivy Tech) - 
Machining (South Central) - 

Informatics (Ivy Tech) - 

Welding (Chaffey) - 

Mechatronics (South Central) - 

Network Infrastructure (Ivy Tech) - 

Software Development (Ivy Tech) - 
Information Technology Support (Ivy Tech) - 
Cyber Security (Ivy Tech) - 

Database Management (Ivy Tech) - 
Computer Science (Ivy Tech) - 

Server Administration (Ivy Tech) - 


Programs 


1 ! 1 ! 
0.0 0.25 0.50 0.75 1.00 
Proportion 


Estimates a Bayesian r) Direct (Wilson) 


E.2.2  TRAINING-RELATED EMPLOYMENT 


The Bayesian procedure made slightly stronger adjustments to the direct estimates for this outcome than 
for program completion, but the estimated between-program variation was large for this outcome as well 
as for program completion, so the adjustments are still mild. Both methods agree on which program is 
best at helping their participants obtain employment related to their training (the Licensed Practical Nurse 
program at Washburn University) and which is worst (the Computer Science program at Ivy Tech). 


As for program completion, several exceptions exist. For example, the Bayesian procedure was much 
more pessimistic than the direct estimates about the chances of future cohorts of participants in the 
Machining program at South Central College. In the other direction, the Bayesian procedure was more 
optimistic than the direct estimates about the chances of future cohorts of participants in five of the 
programs at Ivy Tech (Informatics, Software Development, Database Management, Information 
Technology Support, and Computer Science). Also, the Bayesian procedure provided much tighter error 
bars for six of the programs. 
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Exhibit E-2. Training-related Employment Rate, by Program - Overlaid Bayesian and Direct 
Estimates (Subset to participants not still enrolled at survey follow-up) 


Licensed Practical Nurse (Washburn) - 
Welding Technology (Washburn) - 

Advanced Manufacturing (Manchester) - 
Mechanical Craft (Chaffey) - 

Machining (South Central) - 

Pre-engineering (Chaffey) - 

Industrial Automation (Chaffey) - 

Advanced Manufacturing and Engineering (Chaffey) - 
Certified Production Technician (South Central) - 
Advanced Welding Bootcamp and Program (Bossier) - 
Welding (South Central) - 

Welding Technology (Manchester) - 
Industrial Maintenance (Chaffey) - 

Fast Track to Manufacturing (Bossier) - 
TRAMCON Basic (Miami Dade) - 

Right Skills Now (South Central) - 

CDL and Forklift (Cincinnati) - 

Mechatronics (South Central) - 

TRAMCON Foundation (Miami Dade) - 
Emergency Medical Technician (Washburn) - 
HVAC (Chaffey) - 

TECH 101 (Delgado) - 

Welding (Chaffey) - 

Gateway Courses to IT Programs (Ivy Tech) - 
TRAMCON Advanced (Miami Dade) - 

Core Plus (Delgado) - 

Network Infrastructure (Ivy Tech) - 

Server Administration (Ivy Tech) - 

Cyber Security (Ivy Tech) - 

Informatics (Ivy Tech) - 

Software Development (Ivy Tech) - 

Database Management (Ivy Tech) - 
Information Technology Support (Ivy Tech) - 
Computer Science (Ivy Tech) - 


Programs 


1 1 
0.0 0.25 0.50 0.75 
Proportion 


Estimates a Bayesian ¢ Direct (Wilson) 


E.2.3 PUBLIC ASSISTANCE BENEFIT RECEIPT 


For this outcome, the differences between the Bayesian and direct estimates are much sharper than for 
program completion or training-related employment. For this variable, caterpillar plots demonstrate that 
Bayesian credible intervals are much shorter that the frequentist intervals. Also note that the “spine” of 
the caterpillar is much closer to vertical than would be the case if the programs were sorted by the 
design-based frequentist estimates. Both features are due to the fact that the estimate of between- 
program variance on this outcome was very small. 


Abt Associates Round 4 Early Outcomes Study Report December 2020 1 E-6 
Technical Appendices 


APPENDIX E 


Exhibit E-3. Public Assistance Benefit Receipt, by Program — Overlaid Bayesian and Direct 
Estimates 


Welding (Chaffey) - 

Core Plus (Delgado) - 

Industrial Automation (Chaffey) - 
Pre-engineering (Chaffey) - 

Cyber Security (Ivy Tech) - 

CDL and Forklift (Cincinnati) - 

HVAC (Chaffey) - 

TRAMCON Foundation (Miami Dade) - 

Right Skills Now (South Central) - 

Fast Track to Manufacturing (Bossier) - 
Computer Science (Ivy Tech) - 

Advanced Manufacturing (Manchester) - 
TECH 101 (Delgado) - 

Server Administration (Ivy Tech) - 

Network Infrastructure (Ivy Tech) - 

Industrial Maintenance (Chaffey) - 
TRAMCON Advanced (Miami Dade) - 
TRAMCON Basic (Miami Dade) - 

Welding Technology (Manchester) - 

Licensed Practical Nurse (Washburn) - 
Advanced Welding Bootcamp and Program (Bossier) - 
Software Development (Ivy Tech) - 

Database Management (Ivy Tech) - 
Information Technology Support (Ivy Tech) - 
Mechatronics (South Central) - 

Welding (South Central) - 

Certified Production Technician (South Central) - 
Machining (South Central) - 

Informatics (Ivy Tech) - 

Gateway Courses to IT Programs (Ivy Tech) - 
Welding Technology (Washburn) - 

Advanced Manufacturing and Engineering (Chaffey) - 
Mechanical Craft (Chaffey) - 

Emergency Medical Technician (Washburn) - 


Programs 


0.0 0.2 0.4 0.6 
Proportion 


Estimates a Bayesian ¢ Direct (Wilson) 


E.2.4 CHANGE IN EARNINGS 


The last outcome analyzed is the growth of earnings in two years from three quarters before the program 
start to five quarters after. While the above three outcomes are binary, and are based on self-reports in 
the follow-up survey, the difference in earnings is a continuous variable, and is based on the NDNH 
administrative data. For this outcome, the differences between the Bayesian and direct estimates are also 
quite sharp. The caterpillar plots demonstrate that Bayesian credible intervals are much shorter than the 
frequentist intervals. 
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Exhibit E-4. Change in Earnings, by Program — Overlaid Bayesian and Direct Estimates (Subset 
to participants not still enrolled at survey follow-up) 


Server administration (Ivy Tech) - 

Cyber Security (Ivy Tech) - 

Mechatronics (South Central) - 

Computer Science (Ivy Tech) - 

Network Infrastructure (Ivy Tech) - 
Informatics (Ivy Tech) - 

Machining (South Central) - 

Welding Technology (Washburn) - 

Licensed Practical Nurse (Washburn) - 
Database Management (Ivy Tech) - 

Software Development (Ivy Tech) - 

Welding (South Central) - 

Welding Technology (Manchester) - 
Information Technology Support (Ivy Tech) - 
Mechanical Craft (Chaffey) - 

Right Skills Now (South Central) - 

Advanced Manufacturing (Manchester) - 
Welding (Chaffey) - 

Certified Production Technician (South Central) - 
Gateway Courses to IT Programs (Ivy Tech) - 
Emergency Medical Technician (Washburn) - 
Advanced Manufacturing and Engineering (Chaffey) - 
HVAC (Chaffey) - 

TRAMCON Advanced (Miami Dade) - 
Advanced Welding Bootcamp and Program (Bossier) - 
Fast Track to Manufacturing (Bossier) - 
Industrial Automation (Chaffey) - 

TRAMCON Basic (Miami Dade) - 

Industrial Maintenance (Chaffey) - 
Pre-engineering (Chaffey) - 

TECH 101 (Delgado) - 

TRAMCON Foundation (Miami Dade) - 

CDL and Forklift (Cincinnati) - 

Core Plus (Delgado) - 


Programs 


$-5,000 $0 $5,000 
Change in Earnings 


Estimates & Bayesian $ Direct 


E.2.5 SUMMARY 


Comparing the results qualitatively among the four variables, we can confidently speak about the 
potential strength of Bayesian estimation procedures. While precision gains were modest for the program 
completion and training-related employment, due to high variability of the outcome rates between 
programs, and low predictive power of the models, the estimates generally almost coincided. For public 
assistance benefit receipt and change in earnings, the Bayesian model demonstrated very strong 
predictive power that resulted in substantial precision gains. 


E.3. METHODOLOGY DETAILS 


This final section is highly technical. These details are provided to aid future researchers who may either 
try to reproduce the current results or apply the methodology to similar research projects. 


E.3.1 GENERALIZED LINEAR MIXED MODELS 


For program completion, training-related employment, receipt of public assistance (all based on the 
Follow Up survey data), and growth in earnings (NDNH data), we posit generalized linear mixed models 
that include random effects that account, at least partially, for the idiosyncratic workings of each 
program: 
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Outcome: y,i~f (9 [8,0 

Linear index: 6;; = @ + ee path Aa by Pt 
Mean (inverse link): u;; = E[y;,:|0:] = 9~*(8j), 
Program random effect: u; ~ N(0,t7) 


where j indexes programs, i indexes students within programs, u; is a normally distributed idiosyncratic 
program random effect, and f(-) is an appropriate distribution. For a continuous response (e.g. change in 
earnings), f(-) is usually chosen to be normal (i.e. y;;|9;:,0 ~ N(9;:,07) ) although for the growth in 
earnings (a difference of two highly skewed distributions), a heavier tailed alternative is called for, e.g. 
Vi l9ji,0 ~ Cauchy(6;;,0) with density 


1 


f016,0) = ——— 
Oo 1 + &— 8)" | 


For binary 0/1 outcomes (the primary outcomes based on self-report in the follow-up survey: receipt of 
public assistance benefits, employment in a job related to training, and program completion), the 
appropriate distribution is binomial, most often used with the canonical logit link: 


exp (9) 
EO [yi | | Hyi 1+ exp (9;:) 


(The scale parameter o is not relevant for the binomial distribution.) 


One of the most important parameters of the above model is the variance of the random effects, +. It 
controls the degree of homogeneity of the programs and the extent to which the program-idiosyncratic 
effects impact the program performance. While the goal of the exercise of building a statistical model like 
the above is to improve program-level predictions, larger values of this parameter make this task more 
difficult. In some cases, e.g. in linear models, it can be explicitly shown that, holding everything else 
constant, the larger values of t? lead to a lesser weight that the statistical model has in the resulting 
estimates, and correspondingly a greater weight is being placed on the direct estimates (weighted means 
of outcomes within the program). 

For a set of estimates of the model parameters @, B,7,@ , the model-assisted composite estimate of the 

average outcome 6; for the #th program is: 


Gji 
ol Teed exp( _exp(Gji) _ f (G|{Xji, Vji}) 4G;, binary outcome, 
OY. nj 1+ pi :) 
—— Kyi = 


VG ak ii continuous outcome 
J gj 


6, =@+ Xb, +~ “+X, pjibp + Gi; 
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The integral for the binary outcome is with respect to the posterior distribution of the random effects 
(either empirical Bayes distribution, obtained by plugging the MLE estimates; or the full Bayes when full 
Bayesian estimation is undertaken through MCMC), while for the continuous outcome, the posterior mean 
is used. Although this composite estimate is biased, it generally has smaller mean square error (MSE) 


than the direct estimate y; = —Yi y;, because reduction in variance outpaces increase in (squared) bias. 
J 


Since bias and variance both contribute to the projections of future performance, estimators that 
minimize MSE are generally preferred for forward projections.® If r? is large, the MSE reduction will be 
trivial and the composite estimates will be similar to the direct estimates, but otherwise, the MSE 
reduction can be substantial, particularly for programs with small local sample sizes. Note that 7? is a 
residual variance at the program level, so if the covariates are powerful predictors of the outcome, then 
this variance will be small, and, as a result, the MSE should be reduced for nearly all local programs. 
However, even with small t?, if the local sample size is large, then the difference between the composite 
estimate and the direct estimate will be small. In this case, the variances will also be very similar. 


Intuitively, the variance reduction when both the local sample size and t? are small arises from the fact 


that the “fixed” part of the model (a, B) is estimated on the entire sample rather than just on the local 
sample. This is often referred to as “borrowing strength” across programs. In this case, since student 
outcomes are largely determined by their characteristics at baseline, the mean outcome for any one local 
program is quite accurately estimated by using the outcomes for similar students across all the programs. 
This borrowing of strength tends to “shrink” program-specific estimates toward the average prediction of 
the outcome propensity based on the covariates observed for the participants of the program. (Ina 
simple case of no predictors, the shrinkage is toward the overall mean across all programs.) 


On the other hand, if rt? is large, it is an indication that the variation in student outcomes across 
programs is not explained by the covariates, and that local program features are much more 
determinative than students’ incoming profiles. In this case, borrowing of strength is not helpful and so 
there is minimal shrinkage of local program estimates toward the average fixed-effects-only prediction. 


To further aid intuition about the nature of these model-assisted estimates, note that they can be 
approximately represented as 


6, = $5) fy lr, + (1-4,) > yu My 


iej iej 


The term Yiic; Aji /n; is the model-based estimate of the outcome / completion rate at program 7. While 
reflecting the demographic makeup of the program participants, it does not capture any other 
idiosyncratic features of a program, where idiosyncratic features are those that cannot be explained by 
known factors — in this case the student profile at program entry, the program duration, and the local 
employment rate among the working-age population. By definition, if a program has a particularly good 
workshop for skills practice, a particularly engaging curriculum, or a particularly good instructor, none of 
the positive outcomes due to these factors will be captured in this component. In contrast, the term 
Dies Yji /nj is the direct estimate of the completion rate at the same program. It fully captures the 


8 Rao and Molina (2015, Chapter 5). 
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idiosyncratic features but will generally have much larger variance than the first term, as the sample sizes 
of programs are not particularly large. The compositing factor 


8;(1 — 8) /n, 

_ var(Die; Vj /Nj) _ var(6}, ..., O34) + 8;(1 — 6;)/nj 
~ var(Oy, ..., 34) + var(Die; Vii /Nj) o*/n; 

var(,, ..., 034) + 07/n; 


, binary outcome 


bj 


, continuous outcome 


represents the extent to which information from other programs can be utilized to inform estimation for 


program j. Here, 9; is the true long-run program outcome rate (e.g., completion, or use of public 
assistance) for students at program /if the program continues to be operated under the same general 
conditions. When large sample sizes are available for that program, and/or there is substantial 
idiosyncratic variation across programs, the numerator term is small compared to the population 
between-program variance var(6;, ...,434) Of program-specific outcome rates, and hence the fraction ¢; 
of the model-based estimate in the composite estimate is small, so the composite estimate will be 
approximated equal to the direct estimate for the program. On the other hand, if there is very little 
idiosyncratic variation across programs and if the local sample size is small, n; > 0, the numerator term 
dominates the between-program variance, and ¢; > 1 indicating that the analyst has to rely on the 
model more and more to produce meaningful estimates. 


It is worth noting that throughout the process, the analyst does not interpret the model coefficients 

a, B,t,, does not conduct inference for these, and does not attempt to address their marginal effects on 
the outcome. In other words, the regression model plays only a predictive role. It thus relieves the 
analyst from concerns that are often relevant to structural econometric models, such as endogeneity of 
explanatory variables, omitted variable biases, and correlations between regressors and random effects. 
Whatever helps improving the predictive performance of the model works for SAE. 


Model-assisted small area estimators of this type that blend local (e.g., state, county, school, program, 
class, grantee) experiences with experiences of similar persons in other areas (states, counties, schools, 
programs, classes, grantees) have well established methodologies (Rao and Molina 2015) and are 
common in many federal statistical agencies. Perhaps best known are the school-district level estimates 
of child poverty’ produced by the Census Bureau for the purpose of allocating Title 1 education funds and 
the local estimates of cancer risk factor and screening behaviors!? produced by the National Cancer 
Institute. Procedures like these are also used in evaluations of school and hospital quality, athlete quality 
(in professional sports), and stud quality (in animal science), among other fields.*! Estimators of this sort 
are sometimes referred to as “best” estimators because minimize the mean squared error in the estimate 
about the likely success of future participants rather than narrowly focusing on the experiences of the set 
of participants who happened to attend during the study period and who might have brought 
unmeasured skills and handicaps (i.e., omitted variables) with them. 


9 — https://www.census.gov/programs-surveys/saipe.html [last accessed December 20, 2019] 
10 http://sae.cancer.gov/ [last accessed December 20, 2019] 
4 For an introduction to the many uses of such models, see for example Hox (2010). 
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E.3.2 BAYESIAN ESTIMATION 


Estimating the variance of 6; accurately is extremely difficult with frequentist methods. A common 
approach is to treat the estimated between program variance rt? as known rather than estimated. This 
dramatically simplifies variance estimation, but also leads to systemic underestimation of the variance of 


9 unless the number of programs is very large. Bayesian methods do not suffer from this same defect. 


With them, it is quite easy to estimate the variance of 9% , fully reflecting the extra variance caused by 
uncertainty in the estimate of +”. However, in order to gain this simplification in the variance estimation, 
it is necessary to adopt the entire Bayesian framework in which probabilities are a quantification about 
personal beliefs or a representation of the state of knowledge about the parameters. Users of Bayesian 
methods often try to minimize the influence of personal beliefs by placing extremely vague priors on the 
parameters, going so far as to place improper priors on some of them that say that any real number is 
equally likely. However, it has been demonstrated that improper priors are not an option for t?. A proper 


prior on this parameter is required in order for the posterior distributions of 95 to be proper. 


For the intercept and the regression slopes, improper priors a~1, B,,~1 are usually chosen (leading to 
inference on these parameters that is similar to the frequentist inference). For the variance parameters 
t,o, distributions with support on [0, +00) are usually chosen; popular choices are Gamma distribution 
with very small shape parameters and large scale parameter, log-normal, half-normal, half-Cauchy, and 
uniform over a sufficiently long interval that would cover plausible ranges. 


The most critical prior pertaining to Research Question 8, variability of outcomes between programs, is 
the random effect variance parameter t2. The team chose priors that ran from zero to very large values 
in order to try to minimize the impact of the prior on the final estimates of local program effects. Keeping 
the range with positive support wide should allow the estimate of t? to be dominated by the data rather 
than the preconceptions of the team. 


e For the binary outcomes with logit link, the scale of the linear index 6;; is commensurate with the 


standard deviation of logistic distribution 5 = 1.81: a change of the linear index from 0 to +1.81 


corresponds to a change in probability of the two outcomes from 50%/50% to 14%/86%, which is a 
very drastic change. Thus for the prior distribution of the random effect variance, we used the prior 
that was half-Cauchy(0,1) (i.e., Cauchy distribution restricted to the non-negative numbers): 


2 : 
f@= = eer ae > 0; 0 otherwise 
e For the continuous outcome of change in earnings, given that the quarterly earnings of program 
participants are of the order of tens of thousands of dollars, and one of the primary outcomes, public 
assistance, can be characterized by the federal poverty line of $12,490 for 1-person household, we 
used half-Cauchy(0,$10,000) prior distribution with the scale of $10,000: 
f@M= =;,T > 0; 0 otherwise; s = 10,000 
T 
1S [1 + 5 


The scales of the prior Cauchy distributions are chosen to allow values of t to cover, or to correspond to, 
the expected range of the 6 values (the 99" percentile of the standard Cauchy distribution is 31.82). 
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Later diagnostics show that this distribution is sufficiently noninformative, in that the density of the 
posterior at the posterior mean / mode is much higher than the prior density. 


Bayesian estimates are obtained in the form of posterior distributions. At conceptual level, posterior 
distributions represent application of Bayes theorem: 

Pustetion Gamelan Likelihood(data|parameters) x Prior(parameters) 

Marginal distribution of data 

This formulation produces a continuous distribution that, however, is difficult to impossible to express 
analytically for practical purposes such as computation of the posterior means or credible intervals. Thus 
in practice, to conduct Bayesian inference, the posterior distribution is simulated through sampling. 
Markov chain Monte Carlo (MCMC) methods. These methods generate series of parameter values 
(@, B,7,6) that approximate draws from the joint posterior distribution for the vector of parameters. For 
details, see Gelman et al (2013) and Gill (2014). The fastest implementation of Bayesian methods 
currently available is Stan (http://mc-stan.org; Carpenter et al 2017). It is built upon Hamiltonian Monte 
Carlo (Neal 2011; Monnahan et al 2017), a set of computational techniques that draw upon the dual 
representation of physical systems in terms of “positions” and “moments” that evolve according to the 
laws of Hamiltonian dynamics. 


Explanatory variables X;,,;; used in all the models are listed in Box E-1. To mimic model selection by 
lasso, independent double-exponential, or Laplace, prior distributions were used for the slope parameters 
B, (Kyung et al 2010): 


foo) 


where the mean is zero, and the standard deviation is /2s. For the binary outcomes, the scale s of the 
Laplace prior distributions for regression slopes was set to 0.25. This reflects the prior expectation that 
the variables, if significant, would have odd ratios of about 0.25. This corresponds to the differences in 
actual probabilities of 44% to 56% for the average outcome rate of 50%. For the continuous outcome of 
change in earnings, the scale s of the Laplace prior distributions for regression slopes was set to 2000. 
This corresponds to the prior expectation that the significant variables will likely affect the earning 
changes by + $2,000. The heavy tails of the Laplace distribution, however, allow for much stronger 
effects, while the sharp peak at zero acts to strongly prefer a modal estimate of zero lacking stronger 
evidence of impact of the variable on the outcome. 


E.3.3 BAYESIAN DIAGNOSTICS 


To ensure validity of Bayesian inference based on the simulated MCMC chains, the analyst needs to verify 
that the chains have converged. As it turns out, this is a difficult task. While there are no definitive 
checks, one should expect that the draws should form stationary processes, ideally with minimal to no 
autocorrelation. If there are multiple chains being run, then additionally one would expect that all of them 
converge on the same approximate posterior distributions. As discussed more fully below, we followed 
these practices. We provide the within chain and between chain diagnostics. 
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By default, Stan/RStan reports several measures to diagnose convergence and efficiency of the resulting 
chains. These include: 


e Potential scale reduction factor (Gelman and Rubin 1992) R, defined for multiple chains as the 
square root ratio of the total variance of MCMC draws to the average within-chain variance. If this 
statistic is 1, all chains have converged onto the same set of values. (This is hard to achieve in the 
current project, as each of the multiply imputed data sets implied its own set of posterior 
distributions that differed slightly between imputations.) It is recommended that this statistic is below 


1.10 or 1.05. When only one chain is available, * is computed by splitting the chain in halves, and 
treating the two halves as separate chains. 


e — Effective sample size measures the precision of the posterior mean given the observed 
autocorrelations /,, p>, ... between consecutive draws 8,6), ...,9™) of a parameter 6, relative to 
the ideal uncorrelated sequence. N,-¢ close to the actual number of draws N indicates effective 
simulation, while N.¢¢ « N indicates high autocorrelation which may in turn indicate problems in 
achieving convergence. 


N 


N 
2 
Sj = = 
Nerp = ——=e— Pi = S27 = .. at) — ayet-) —@ 
ff Thay Pi Pe Sj ( )¢ ) 


t=l41 


The actual formulae used by RStan for both & and Negr differ slightly by including various small sample 
adjustments that improve performance of these measures with few and shorter chains. 


Hamiltonian Monte Carlo has its own additional sets of tuning parameters and computational techniques 
to improve accuracy and convergence. In this project, Hamiltonian MCMC was run with the following 
tuning parameters: 


e adapt_delta = 0.96, elevated from the default value of 0.8. This is the primary parameter that should 
be adjusted in Hamiltonian MCMC. Higher values (those closer to 1) of this parameter force the 
algorithm to make smaller steps thus improving the quality of numeric approximations of the 
Hamiltonian trajectories. While simple regression models converge fine with the default setting of 
0.8, the current model is more complicated in two aspects. First, it uses random effects that 
effectively introduce correlations between individual observations in the data set. Second, it uses 
non-differentiable Laplace priors that generally require very small steps and high values of the delta 
parameter. !2 


e stepsize_jitter = 0.2. Jitter of step sizes is not necessary, but it helps avoid (very infrequently 
encountered) cyclical behaviors of chains. 


e max_treedepth = 12, elevated from the default value of 10. The No-U-Turn Sampler (NUTS) 
implemented in Stan utilizes a physical principle of Hamiltonian dynamics that a system is bound to 
return to its state after a certain period of time, and evaluates the number of steps until the U-turn 
by attempting 1, 2, 4, 8, etc. steps, doubling the number of steps until the U-turn is encountered. 
This parameter caps the number of such duplications, thus limiting the number of steps along each 


12 In his tutorial on Bayes sparse regression, one of Stan developers used an even higher value of 0.99, although in his 
example, the number of predictors, 200, exceeded the number of observations, 100. See 
https://betanalpha.github.io/assets/case_studies/bayes sparse _regression.html#34 laplace_prior. 
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trajectory by 2™2* -treedepth _ 4, Tt should be set in such a way that in estimation runs, the tree depth 
is never saturated. 


e Warmup period = 1000 iterations, estimation period = 1000 iterations. The warmup period is used to 
calibrate the parameters of Hamiltonian Monte Carlo, in particular the step sizes. (Small step sizes 
produce better approximations at the expense of slower computing; larger step sizes run the risk of 
approximations being so poor that the “energy” of the underlying physical system blows up to 
infinity, which must be avoided, e.g. by choosing a different random direction for a trajectory from 
the current MCMC draw.) The estimation period is used to produce the posterior draws. It is crucial 
that the chains shall have converged by the beginning of the estimation period. 


An appropriate combination of parameters is obtained by trial and error. While demonstrating that the 
chains have converged is mathematically impossible, diagnostics of failure of convergence are more 
readily available. Beyond the diagnostics developed for other types of MCMC, such as examination of 
autocorrelations, traceplots, Gelman-Rubin convergence diagnostics, etc., Hamiltonian MCMC 
implemented in RStan provides additional diagnostics such as frequency of divergent transitions and 
changes in the chain “energy.” A successful transition from the warmup to the estimation period in 
Hamiltonian MCMC can also be tracked by (1) lack of divergent steps, (2) stabilization of step size, 

(3) tree depth comfortably below the max depth. All of these behaviors were checked and observed in 
this project. 


One MCMC chain was run per each of the M=12 imputed data sets. For each chain, and for the collection 
of chains, the Bayesian MCMC diagnostics were checked. Diagnostics specific to Hamiltonian MCMC 
included making sure that, in the estimation part of the chain, there are no divergent transitions (a 
transition is declared divergent if there is evidence that approximation of a trajectory becomes 
inaccurate), that mean acceptance statistic is about the same as the adaptive delta parameter, and that 
the treedepth never achieves its maximum. All of the chains passed these checks. 


For each given imputed data set, the chain corresponding to that data set demonstrated convergence 
and expected performance, as gauged by the variance reduction factors R-hat being less than 1.05. 


A rare exception is the random effects standard deviation t parameter in estimation of the public 
assistance outcome: three out of twelve chains had R-hat values in excess of 1.1, namely 1.48, 1.17 and 
1.13. This parameter also exhibited very high autocorrelations, with at least ten lags having 
autocorrelations greater than 0.1. Traceplots of t indicated the reason: sometimes the chain would get 
stuck with values of t very near zero that are difficult to escape. This did not appear to be only a warmup 
length issue, as chains would stray into values near zero from apparently stable periods where their 
performance would be just as good as of other chains that never visited zero. This issue however may 
not be particularly problematic for this outcome: since the regression model included the primary 
determinants of eligibility for public assistance, income and household composition, the fixed portion of 
the model had very high explanatory power. It is not then surprising that the remaining variation 
between programs was poorly identified. 


When the chains were combined in order to conduct joint inference that would account for the multiple 
imputation, the overall posterior summaries indicated that R-hat statistics would deteriorate very 
substantially, with about a quarter of the variables having R-hats reported as above 1.05. This strong 
interaction with multiple imputation is not surprising, and reflects the between-imputation variability that 
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is intrinsic, and is in fact desirable, for this missing data method. Put differently, each chain is estimating 
its own parameter corresponding to that specific imputation, and different chains center on slightly 
different parameter values. 


Upon establishing adequate quality of Bayesian posterior draws, we computed the program-specific 
outcome rates 9 for each draw. The summaries (means and distribution percentiles) of these composite 
estimates are reported in Appendix G and visualized using caterpillar plots in Chapter 7. Despite 
variations between chains, R-hat values were below 1.06 for all program-specific outcome estimates. 


E.4. TECHNICAL DISCUSSION OF FITTED MODELS UNDERLYING 
LOCAL ESTIMATES 


This section presents the numeric diagnostic summaries and plots of priors and posteriors for key 
parameters, the random effect variance, and one of the regressors, by outcome. 


The prior distribution for regression coefficients was double exponential / Laplace which has a sharp peak 
at zero and tails heavier than the normal distribution. This prior distribution was chosen because the 
posterior mode in regression models corresponds to the frequentist lasso estimate. The prior distributions 
of the random effect variances were half-Cauchy, i.e., the positive values of the Cauchy distribution with 
the location parameter of zero and scale parameter of 1. Random effect variance directly address the 
issue of variability of outcome rates, i.e., Research Question 8. However, if the random effect models 
cannot extract information concerning this variability, then this Research Question can scarcely be 
answered. Hence we hope to see that the posterior distribution is much more concentrated than the prior 
distribution. Graphical displays will also be supplemented by Bayesian computing diagnostics to ensure 
that we can appropriately interpret the results. For both types of parameters, we hope to see that the 
posterior is more concentrated that the prior. 


E.4.1 PROGRAM COMPLETION 


The following Bayesian summaries apply to the random effect standard deviation parameter t: 


e Posterior mean = 1.15 

e Posterior standard deviation = 0.18 

e 95% highest posterior density credible interval = (0.85; 1.55) 
e Potential scale reduction factor Rho = 1.00 


e Effective sample size = 7,196 (out of 12,000) 
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Exhibit E-5 displays autocorrelations for estimates of t by lags between draws. There is one panel for 
each of the first five chains. The other chains performed similarly. Note that the auto correlation for 
lag=1 (consecutive draws) is about 0.1 and climbs slightly for lag=2 for some unknown reason. 
Autocorrelations for lag=3 and higher is near zero, which is a good thing. 


Exhibit E-5. Autocorrelation in Estimates of Between-Program Standard Deviation in Program 


Completion Rate by Lag 
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Exhibit E-6 displays both the prior and posterior densities for the between-program standard deviation 
in the program completion rate. This is the critical parameter that governs the degree of shrinkage. The 
dashed line shows the half-Cauchy distribution used as the prior distribution. It has the highest density at 
t = 0, and a very gradually declining density for larger values. The solid line shows the posterior 
distribution. Note the sharp peak at about t = 1.2 and note how the density fall to zero for t < 0.7 and 
for t > 1.8. This indicates that the observed data support the belief that there is very large variation in 
program completion rates even after accounting for student profiles and program duration. (Recall that 
this standard deviation is on the logit scale, so a value of 1.2 is very large.) Because of this large value 
for t, the Bayesian procedure estimated program-specific program completion rates very similar to those 
estimated by the design-based frequentist method, as shown in Exhibit E-1. 


Exhibit E-6. Overlaid Prior and Posterior Densities for Between-Program Standard Deviation in 
Program Completion Rate 
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Exhibit E-7 shows a similar prior-vs-posterior graph for the effect of being under age 21 on the logit 
probability of program completion. (There is nothing special about this particular variable—it was picked 
merely for illustrative purposes.) Prior vs. posterior plot shows that the prior is strongly informative, as 
the posterior has about the same height and spread as the prior. However the information contained in 
the data shifted the distribution; the parameter can be considered “significant” since the 95% credible 
interval does not cover zero. 


Exhibit E-7. Overlaid Prior and Posterior Densities for Effect of Being under Age 21 on Program 
Completion Rate 
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E.4.2 TRAINING-RELATED EMPLOYMENT 
The following Bayesian summaries apply to the random effect standard deviation parameter t: 
e Posterior mean = 0.98 
e Posterior standard deviation = 0.17 
e 95% highest posterior density credible interval = (0.70; 1.37) 


e Potential scale reduction factor Rho = 1.01 


e Effective sample size = 5,246 (out of 12,000) 
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Exhibit E-8 displays both the prior and posterior densities for the between-program standard deviation 
in the training-related employment rate. This is the critical parameter that governs the degree of 
shrinkage. The dashed line shows the half-Cauchy distribution used as the prior distribution. It has the 
highest density at t = 0, and a very gradually declining density for larger values. The solid line shows the 
posterior distribution. Note the sharp peak at about t = 0.85 and note how the density fall to zero for t < 
0.5 and for t > 1.7. This indicates that the observed data support the belief that there is very large 
variation in training-related employment rates even after accounting for student profiles and program 
duration. This is a bit smaller than for program completion, but still very large. Because of this large 
value for t, the Bayesian procedure estimated program-specific training-related employment rates very 
similar to those estimated by the design-based frequentist method, as shown in Exhibit E-2. 


Exhibit E-8. Overlaid Prior and Posterior Densities for Between-Program Standard Deviation in 
Training-Related Employment Rate 
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Exhibit E-9 shows a prior-vs-posterior plot for the effect of being under age 21 on the logit probability 
of job alignment. Prior vs. posterior plot shows that the prior is strongly informative, as the posterior has 
about the same height and spread as the prior. Information added by the data is not sufficient to move 


the posterior distribution away from zero. 


Exhibit E-9. Overlaid Prior and Posterior Densities for Effect of Being under Age 21 on Training- 
Related Employment Rate 
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E.4.3 PUBLIC ASSISTANCE BENEFITS RECEIPT 
The following Bayesian summaries apply to the random effect standard deviation parameter t: 


e Posterior mean = 0.17 

e Posterior standard deviation = 0.11 

e 95% highest posterior density credible interval = (0.01; 0.41) 

e Potential scale reduction factor Rho = 1.11 

e Effective sample size = 97 (out of 12,000) 

Convergence was less than satisfactory for this parameter. However, this is not particularly problematic 


for interpretation of Bayesian estimates, since the estimate itself is small. This in turn is due to high 
explanatory power of fixed effects that used receipt of public assistance at baseline as predictors that 
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have very strong explanatory power: the coefficient for receipt of SNAP has a 95% posterior interval of 
(1.62, 2.33), and receipt of TRA, (0.49, 2.04). We alluded to this situation in our explanation of the 
allocation of the explanatory power within the statistical model: once most of the variability of outcomes 
is explained by the fixed effects portion of the model, the residual variance to be explained is small and 
more difficult to identify. 


Exhibit E-10 displays both the prior and posterior densities for the between-program standard deviation 
in the public assistance benefits receipt rate. This is the critical parameter that governs the degree of 
shrinkage. The dashed line shows the half-Cauchy distribution used as the prior distribution. It has the 
highest density at t =0, and a very gradually declining density for larger values. The solid line shows the 
posterior distribution. Note that this posterior distribution is not as sharply peaked as those in 

Exhibits E-6, E-8, and E-9. There is an apparent accumulation of density at + =0. A value of zero 
would indicate that there is no idiosyncratic variation in this outcome after controlling for the student 
profile, program duration, and the local employment rate. But fairly large values of t cannot be ruled out. 
All in all, it appears that the data on this outcome provide less information for estimation of the t 
parameter than for other outcomes, as evidenced by a low effective sample size (not shown). Because of 
the small value for t, the Bayesian procedure estimated program-specific public assistance receipt rates 
are very different from those estimated by the design-based frequentist method, as shown in 

Exhibit E-4. 


Exhibit E-10. Overlaid Prior and Posterior Densities for Between-Program Standard Deviation 
in Rate of Public Assistance Benefits Receipt 
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Exhibit E-11 shows a prior-vs-posterior plot for the effect of being under age 21 on the inverse logit 
probability of public assistance benefits receipt. Prior vs. posterior plot shows that the prior is strongly 
informative, as the posterior has about the same height and spread as the prior. The estimate is 
“significant” as the 95% highest posterior density credible interval (-0.805, -0.013) is isolated from zero. 


Exhibit E-11. Overlaid Prior and Posterior Densities for Effect of Being under Age 21 on Rate of 
Public Assistance Benefits Receipt 
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E.4.4 CHANGE IN EARNINGS 


The following Bayesian summaries apply to the random effect standard deviation parameter t: 


e Posterior mean = $599 

e Posterior standard deviation = $278 

e 95% highest posterior density credible interval = ($78.67, $1,210) 
e Potential scale reduction factor Rho = 1.04 


e Effective sample size = 313 (out of 12,000) 
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Convergence was less than ideal for this parameter. As was the case for the public assistance receipt, 
reduction in the explained variability in the outcome is driven by the explanatory variables in the outcome 
regression, with e.g. the age predictors and program duration having strong significant effects: age 20 or 
less, 95% posterior credible interval of ($1,123; $2,593); age 21 to 24, 95% posterior credible interval of 
($15; $1,348); program length in years, 95% posterior credible interval of ($409; $2,322). 


Exhibit E-12 displays both the prior and posterior densities for the between-program standard deviation 
in the change in earnings. The dashed line shows the half-Cauchy distribution used as the prior 
distribution. The data shifts the distribution very significantly. Between-program variability is lower than 
both the effects of the strongest predictors quoted in the preceding paragraph, and residual variability 
(Exhibit E-13) which is about 5 times greater, and has a 95% posterior credible interval of ($2,390; 


$2,744). 


Exhibit E-12. Overlaid Prior and Posterior Densities for Between-Program Standard Deviation 
in Change in Earnings 
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Exhibit E-13. Overlaid Prior and Posterior Densities for Residual Standard Deviation in Change 


in Earnings 
Prior and posterior residual variance 


0.004 - 
0.003 - 
a> 
c —] Posterior 
T 0.002-  -! Prior 
mal 
0.001 - 
0.000 - 
0 1000 2000 3000 
sigma 


Abt Associates Round 4 Early Outcomes Study Report December 2020 1] E-25 
Technical Appendices 


APPENDIX E 


Exhibit E-14 shows a prior-vs-posterior plot for the effect of being under age 21 on the change in 
earnings. Prior vs. posterior plot shows that the prior is weakly informative, as it has lower height and 
greater spread compared to the posterior, but is still noticeable unlike say the prior in Exhibit E-13. The 
estimate is “significant” as the 95% highest posterior density credible interval ($1,123; $2,593) is isolated 
from zero. 


Exhibit E-14. Overlaid Prior and Posterior Densities for Effect of Being under Age 21 on Change 
in Earnings 
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F. Implementation Data Collection 


This appendix describes implementation data collection. Section F.1 discusses the site visits conducted by 
the research team and Section F.2 describes how the site visit data were analyzed. 


Fil. SITE VISITS 


The research team conducted site visits to the nine outcome study grantees in March and April 2017. The 
site visits were designed to inform the outcomes study analysis, documenting participant experiences in 
the training programs and grantees’ implementation of capacity-building activities. The site visits were 
intended to contextualize findings from the outcomes study. 


The research team’s site visitors worked with grant staff to schedule the site visits. > Site visitors typically 
worked with the grant director to set site visit dates and to develop an agenda for the visit, including the 
colleges and campuses to be visited and staff to be interviewed. The length of the visits varied from one 
grantee to the next based on the number of programs that were included in the outcomes study and the 
number of colleges (for consortiums) and campuses (for consortiums and single institution grantees) to 
be visited. In some cases, it was not possible to visit each college or campus. In those situations, site 
visitors prioritized visiting colleges with several programs in the outcomes study over colleges that had 
only one program in the study. If multiple colleges in a consortium were offering similar versions of the 
same program, the site visitors visited a subset of the colleges. 


Site visitors conducted semi-structured interviews with four types of stakeholders: college leadership, 
grant directors, key partners and employers, and faculty and staff.'* Interviews typically lasted between 
one and one and a half hours. Site visitors took verbatim notes and made audio recordings of the 
interviews. Interview guides were designed to collect data on the grant’s institutional context; planning 
and initial implementation; recruitment; key components of training programs; capacity-building 
activities; partnerships and systems alignment efforts; and post-grant plans and sustainability. 


The research team also toured renovated training facilities and classrooms. When possible, the team 
attended a session of the training programs and took notes in an observation tool. The observation tool 
captured data on the setting and physical environment, content of the session, teaching methods, 
participant engagement, participant-instructor rapport, and participant relationships. 


After each site visit, site visitors reviewed their notes to clarify any ambiguities and resolve typos or other 
errors. Site visitors then developed a report to summarize key findings. 


13 Site visitors consisted of one senior and one junior member of the research team. The senior member led the interviews 
and the junior member scheduled the visit, planned logistics, and took notes. Two site visit teams visited the Chaffey 
College consortium and Ivy Tech College because both grantees had multiple colleges or campuses and many training 
programs included in the outcomes study. 


14 Interview guides can be found at https://www.reginfo.gov/public/do/PRAICList?ref_nbr=201505-1291-001. 
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F.2. ANALYSIS 


The research team developed a set of spreadsheets to capture and analyze information from the site visit 
reports. The categories in the spreadsheet corresponded to the topics that are highlighted in the final 
report. The categories were accelerated and enhanced learning, persistence and completion, connections 
to employment, partnerships, successes and challenges, and sustainability. The research team reviewed 
and analyzed the tables, noting themes that emerged across the site visits and highlighting examples of 
illustrative activities and practices. This process of review and analysis formed the basis for the material 
included in the report. 
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APPENDIX G 


This appendix provides expanded results for the analyses presented in Chapters 4 through 7 of this 


report. 


G.1. EXPANDED RESULTS FOR CHAPTER 4 


Exhibit G-1 contains expanded results of the participant characteristics reported at program entry, 
including the mean and 95 percent confidence interval for each measure. 


Exhibit G-1. Expanded Results, Participant Characteristics at Program Entry 


Characteristic | Mean 95% Confidence Interval Sample Size 

Age (%) 

20 or younger 23.0 (17.4, 28.6) 2,767 

21 to 24 17.5 (15.4, 19.6) 2,767 

25 to 34 30.2 (27.6, 32.8) 2,767 

35 or older 29.4 (24.3, 34.5) 2,767 
Gender (%) 

Male 85.4 (79.8, 91.0) 2,767 

Female 14.6 (9.0, 20.2) 2,/67 
Race/Ethnicity (%) 

Hispanic, any race 21.6 (12.6, 30.5) 2,767 

White (Non-Hispanic) 47.7 (36.1, 59.2) 2,767 

Black (Non-Hispanic) 22.5 (10.2, 34.8) 2,767 

Asian (Non-Hispanic) 4.2 (2.6, 5.8) 2,767 

Other, including multi-race (Non-Hispanic) 4.0 (3.3, 4.8) 2,767 
Family Structure (%) 

No spouse/partner, no child 56.0 (51.6, 60.3) 2,767 

No spouse/partner, has children 14.2 (11.8, 16.7) 2,767 

Has spouse/partner, no children 13.4 (12.1, 14.8) 2,767 

Has spouse/partner, has children 16.4 (13.0, 19.8) 2,767 
Education Attainment (%) 

Less than high school 4.1 (2.0, 6.1) 2,767 

High school or GED 42.8 (37.3, 48.3) 2,767 

Technical or vocational degree 6.0 (4.8, 7.3) 2,767 

Some college but no degree 32.0 (28.6, 35.3) 2,767 

Associate’s degree 7.4 (5.7, 9.1) 2,767 

Bachelor’s degree or more 7.7 (5.5, 10.0) 2,767 
Family Income in past 12 months (%) 

$0 - $14,999 30.3 (25.0, 35.6) 2,767 

$15,000 - $29,999 27.6 (25.3, 29.9) 2,767 

$30,000+ 42.1 (36.5, 47.7) 2,767 
Housing (%) 

Own the place where you live 17.9 (15.6, 20.2) 2,767 

Rent own place or contribute to rent at friend 53.2 (49.1, 57.2) 2,767 

or family’s place 

Live rent free 28.9 (25.2, 32.6) 2,767 
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Characteristic Mean 95% Confidence Interval Sample Size 

Public Assistance (%) 

Currently receiving TRA 1.6 (0.7, 2.5) 2,767 

Currently receiving SNAP 11.5 (8.3, 14.6) 2,767 

Currently receiving TANF 1.6 (0.8, 2.4) 2,767 

Currently receiving TANF or SNAP 11.8 (8.6, 15.0) 2,767 

Currently receiving UI 5.3 (2.7, 7.9) 2,767 
Employment Status (%) 

Currently working 56.4 (51.8, 60.9) 2,767 

Not currently working, but worked at some 

point sunathe ae months ae shy) ao! 

Not currently working, longer than 12 

months ae last aes ; M2 (et, 12.3) ie 

Never worked 3.7 (3.0, 4.4) 2,/67 
Expected Work Hours (%) 

0 22.4 (18.5, 26.3) 2,767 

1 to 19 7.3 (5.0, 9.7) 2,767 

20 to 34 23.1 (18.9, 27.2) 2,767 

35 and plus 47.2 (38.8, 55.5) 2,767 
Citizenship and Veteran Status (%) 

Citizen and veteran 8.9 (6.9, 10.9) 2,/67 

Citizen, not veteran 84.5 (81.9, 87.1) 2,/67 

Not citizen 6.6 (4.2, 8.9) 2,/67 
Language (%) 

Speak English at home 74.2 (66.7, 81.7) 2,767 

Speak language other than English at home 25.8 (18.3, 33.3) 2,767 
Expected Program Participation (%) 

Expect to participate full time 59.5 (48.4, 70.6) 2,767 

Expect to participate part time 40.5 (29.4, 51.6) 2,767 
Most Important Reasons to Participate (%) 

Find Work 19.5 (16.5, 22.6) 2,767 

Career Change 25.2 (21.9, 28.5) 2,767 

Career Advancement 29.2 (25.0, 33.4) 

Education Advancement 14.4 (11.7, 17.2) 2,767 

Personal Reasons 8.3 (7.0, 9.7) 2,767 

Others 3.4 (2.6, 4.2) 2,767 
Prior Experience in Industry (%) 

None 60.4 (54.5, 66.2) 2,767 

Some but less than one year 6.0 (4.8, 7.1) 2,767 

One year or more 33.7 (28.2, 39.2) 2,767 
Training Duration (%) 

Greater than one semester 45.7 2,767 
Household Income below Poverty Line (%) 

Household is below poverty level 42.5 (37.0, 48.1) 2,767 
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Exhibits G.2 to G.5 contains expanded results of the outcomes reported in Chapter 4, including training 
duration, service receipt, training, and employment, earnings, and income. 


Exhibit G-2. Expanded Results, Training Duration Outcomes 


Outcome Mean 95% Confidence Interval Sample Size 
Training duration, full sample 
Total months of training 7.4 (6.5, 8.3) 2,211 
FTE months of training 6.2 (5.4, 7.0) 2,211 
More than 6 FTE months of training (%) 44.2 (34.2, 54.3) 2,211 


Training duration, among those who 
finished classes 


Total months of training 6.2 (5.2, 7.2) 1,313 
FTE months of training 5.5 (4.4, 6.6) 1,313 
Six or more FTE months of training (%) 38.8 (23.9, 53.8) 1,313 


Exhibit G-3. Expanded Results, Service Receipt Outcomes 


Outcome Mean 95% Confidence Interval Sample Size 
Accelerated Learning (%) 
Received transfer credits earned from previous 20.7 (13.9, 27.5) 2,211 
college 
Number of transfer credits received 5.6 (4.0, 7.1) 459 
Received transfer credits earned from prior 
learning 4.4 (3.2, 5.5) 2,211 
Number of prior learning or work experience 
credits received iA (0.9, 2.0) a3 
Stated there is a recommended program leading 
to next level credential es ee a2l4 
Completion and Persistence (%) 
Attended course focusing on study skills, 
workplace skills, or general life skills nap (0:o ae) eek 
Ever took career planning course 22.1 (18.2, 26.0) 2,211 
Ever took course about study skills 23.6 (19.5, 27.6) 2,211 
Ever took a course about job search 20.7 (17.2, 24.2) 2,211 
Ever took a course about critical thinking and 
problem solving 27.5 (23.7, 31.2) 2,211 
Ever took a course about finding help with life 
problems 19.0 (15.8, 22.2) 2,211 
Ever took a course about financial aid for school 15.8 (12.4, 19.3) 2,211 
Ever took a course about time management 22.0 (18.5, 25.6) 2,211 
Ever took a course about working in groups 24.6 (20.7, 28.6) 2,211 
Ever took a course about communicating well 26.2 (21.8, 30.6) 2,211 
pias a course about managing stress and 11.0 (8.2, 13.8) 2,211 
Ever took a course about staying motivated 21.9 (17.9, 25.8) 2,211 
Ever took a course about acting professionally 28.6 (24.1, 33.0) 2,211 
Ever took a course about managing finances 9.9 (6.5, 13.3) 2,211 
Ever took a course about handling family 
responsibilities ou (410.8) Zens 
Received academic advising 47.8 (40.4, 55.1) 2,211 
Received financial aid advising 31.8 (24.3, 39.4) 2,211 
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Outcome Mean 95% Confidence Interval Sample Size 
Sources of Financial Assistance for Training 
Programs (%) 
Own earnings or savings or those of a spouse 49.5 (45.6, 53.4) 2,211 
Loans in own name or name of a family member 21.0 (13.5, 28.5) 2,211 
A parent or other family member 25.4 (21.4, 29.4) 2,211 
Grant from the government 38.8 (31.6, 46.1) 2,211 
Used TRA benefits to pay for training costs 7.6 (4.1, 11.2) 2,211 
Veteran's benefits 5.2 (3.4, 7.0) 2,211 
Scholarship 14.4 (9.1, 19.8) 2,211 
Financial support from employer 9.2 (6.1, 12.2) 2,211 
Other funding source 15.6 (13.0, 18.2) 2,211 
Work-based Learning (%) 
Coursework utilized virtual workplace simulations 
to practice 48.0 (38.2, 57.8) 2,211 
Coursework utilized work-like physical 
environments to practice cs 0:01 03:8) eer 
Offered arranged visits from employer/learning 
about eriployars 51.4 (43.2, 59.6) 2214 
Offered class taught by instructors from local 
employer/class 35.3 (30.3, 40.2) 2,211 
Opportunity for work study job or internship 56.7 (47.4, 66.1) 2,211 
Offered a work study job as part of studies 26.8 (19.7, 34.0) 2,211 
Offered clinical experience or practicum as part 
af chidics 21.7 (14.3, 29.1) 2,211 
Offered an apprenticeship as part of studies 22.1 (15.2, 28.9) 2,211 
Offered an internship as part of studies 25.0 (14.2, 35.8) 2,211 
Offered other work experience as part of studies 29.5 (25.0, 34.1) 2,211 
Received career counseling 34.8 (30.7, 38.9) 2,211 
Received job search or placement assistance 39.5 (31.2, 47.8) 2,211 
Create or edit a resume 32.8 (24.7, 40.9) 2,211 
Look for a job 33.3 (25.5, 41.2) 2,211 
Use web-based search engines 23.0 (17.5, 28.5) 2,211 
Find specific job leads 29.8 (22.7, 37.0) 2,211 
Fill out job applications 20.3 (15.5, 25.2) 2,211 
Opportunity for coached job interview practice 25.4 (18.1, 32.7) 2,211 
Academic Advising (%) 
Only college provided academic advising 90.0 (87.0, 93.0) 1,058 
Only other organization provided academic 
advising 4.0 (2.2, 5.9) 1,058 
Both college and other organization provided 
academic advising e0 ae, 77) 1,058 
Financial Aid Advising (%o) 
Only college provided financial advising 87.1 (83.8, 90.4) 696 
Only other organization provided financial 
advising 6.1 (3.4, 8.8) 696 
Both college and other organization provided 
financial advising 6.8 (4.4, 9.1) 696 
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Outcome Mean 95% Confidence Interval Sample Size 

Career Counseling (%) 
Only college provided career counseling 85.2 (81.4, 89.0) 768 
Only other organization provided career 5.3 (3.0, 7.5) 768 
counseling 
Both college and other organization provided 96 (6.8, 12.3) 768 
career counseling 

Job Search Assistance (%) 
Only college provided job search assistance 79.2 (72.2, 86.2) 872 
Only other organization provided job search 73 (2.7, 11.8) 872 
assistance 
Both college and other organization provided job 13.5 (9.7, 17.3) 872 
search assist 

Resume Help (%) 
Only college provided resume help 78.6 (72.3, 84.9) 722 
Only other organization provided resume help 5.5 (1.9, 9.1) 722 
Both college and other organization provided 15.9 (11.8, 19.9) 322 
resume help 

Job Search Training (%) 
ce college provided help on how to look for 74.2 (69.0, 79.5) 735 
Only other organization provided help on how to 43 (1.5, 7.2) 735 
look for job 
Both college and other organization provided 
help on how to look for job a Gite) 78 

Job Search using Search Engine (%) 
Only college provided help using job search 75.5 (69.7, 81.4) 507 
engines 
Only other organization provided help using job 
search engines 6.3 (2.6, 10.0) 507 
Both college and other organization help using 18.2 (14.7, 21.6) 507 
job search engines 

Finding Specific Job Leads (%) 
i a provided help finding specific job 76.0 (70.1, 81.9) 657 
Only other organization provided help finding 48 (1.9, 7.7) 657 
specific job leads 
Both college and other organization help finding 19.2 (14.7, 23.7) 657 
specific job leads 

Job Applications (%) 
Only college provided help filling out job 73.3 (64.6, 82.1) 445 
applications 
Only other organization provided help filling out 78 (2.4, 13.2) 445 
job applications 
Both college and other organization help filling 18.9 (13.7, 24.0) 445 
out job applications 

Job Interviews (%) 
Only college provided help practicing for job 
aS teMacie 78.3 (71.2, 85.4) 557 
oa other organization provided help practicing 72 (2.4, 12.0) 557 
or job interviews 
Both college and other organization help 14.5 (10.5, 18.5) 557 
practicing for job interviews 
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Outcome Mean 95% Confidence Interval Sample Size 
Satisfaction with Program (%) 
Very satisfied with program 59.8 (55.6, 64.0) 2,211 
Somewhat satisfied with program 32.8 (29.1, 36.6) 2,211 
Not satisfied with program 7.5 (6.2, 8.8) 2,211 


Exhibit G-4. Expanded Results, Training Outcomes 


Outcome Mean 95% Confidence Interval Sample Size 

Finished Classes (%) 59.4 (49.9, 68.8) 2,211 
Left without Finishing Classes (%) 23.5 (18.7, 28.4) 2,211 
reas Enrolled in Required Classes at Follow-up 17.1 (10.6, 23.6) 2,211 
Credential (%) 

Finished classes and received any credential 51.4 (40.8, 61.9) 2,211 

Finished classes and received target credential 28.7 (18.1, 39.4) 2,211 
Additional Training (%) 

Finished classes and started additional training 16.6 (11.5, 21.7) 2,211 

Finished classes, received any credential, and 

started additional training 14.4 (9.4, 19.4) 2214 

Finished classes, received target credential, and 

started additional training ae (6.6; 12:6) oer 
Return to College (%) 

Plan to return to college 52.1 (48.1, 56.1) 2,211 

Plan to return to college, completed program 26.1 (19.5, 32.7) 2,211 

Plan to return to college, still enrolled in program 10.9 (6.8, 15.0) 2,211 

Plan to return to college, left program 15.1 (11.3, 18.9) 2,211 

Number of months until planning to return to 

college 8.3 (6.9, 9.7) 1,154 
College Credits (%) 

Earning college credits 32.2 (20.8, 43.6) 2,211 

Total college credits earned since beginning 16.9 (13.4, 20.4) 711 

program 
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Exhibit G-5. Expanded Results, Employment, Earnings, and Income Outcomes 


felTi gee) iit= | Mean 95% Confidence Interval Sample Size 

Employment (%) 
Employed in 7th quarter before program entry 63.0 (58.8, 67.3) 1,845 
Employed in 6th quarter before program entry 64.1 (60.3, 67.8) 2,355 
Employed in 5th quarter before program entry 66.1 (62.0, 70.2) 2,355 
Employed in 4th quarter before program entry 68.1 (63.7, 72.5) 2,355 
Employed in 3rd quarter before program entry 71.1 (67.3, 74.9) 2,355 
Employed in 2nd quarter before program entry 70.6 (67.1, 74.1) 2,355 
Employed in ist quarter before program entry 71.3 (67.6, 75.1) 2,355 
Employed in quarter of program entry 66.2 (61.7, 70.6) 2,355 
Employed in ist quarter after program entry 69.3 (65.4, 73.2) 2,355 
Employed in 2nd quarter after program entry 75.0 (71.8, 78.2) 2,355 
Employed in 3rd quarter after program entry 77.8 (74.2, 81.5) 2,355 
Employed in 4th quarter after program entry 76.4 (71.2, 81.7) 2,355 
Employed in 5th quarter after program entry 77.5 (73.3, 81.8) 2,355 

Number of quarters employed between ist and 

5th quire (range 0 to 5). ae (3.59, 3.93) 2,390 

Earnings ($) 
Earnings in 7th quarter before program entry 4,276 (3,786, 4,766) 1,845 
Earnings in 6th quarter before program entry 4,155 (3,655, 4,656) 2,355 
Earnings in 5th quarter before program entry 4,434 (3,894, 4,974) 2,355 
Earnings in 4th quarter before program entry 4,787 (4,203, 5,372) 2,355 
Earnings in 3rd quarter before program entry 4,915 (4,304, 5,526) 2,355 
Earnings in 2nd quarter before program entry 5,099 (4,465, 5,733) 2,355 
Earnings in 1st quarter before program entry 4,888 (4,132, 5,645) 2,355 
Earnings in quarter of program entry 4,452 (3,638, 5,266) 2,355 
Earnings in 1st quarter after program entry 4,602 (3,776, 5,428) 2,355 
Earnings in 2nd quarter after program entry 5,420 (4,650, 6,191) 2,355 
Earnings in 3rd quarter after program entry 6,149 (5,374, 6,924) 2,355 
Earnings in 4th quarter after program entry 6,750 (5,682, 7,819) 2,355 
Earnings in 5th quarter after program entry 7,187 (6,136, 8,238) 2,355 

Earnings ($), among those employed in a job 

related to training 
Earnings in 7th quarter before program entry 5,363 (4,591, 6,135) 371 
Earnings in 6th quarter before program entry 5,446 (4,686, 6,207) 459 
Earnings in 5th quarter before program entry 5,808 (4,874, 6,743) 459 
Earnings in 4th quarter before program entry 5,866 (4,864, 6,869) 459 
Earnings in 3rd quarter before program entry 6,038 (4,942, 7,135) 459 
Earnings in 2nd quarter before program entry 6,100 (5,173, 7,028) 459 
Earnings in 1st quarter before program entry 5,601 (4,738, 6,464) 459 
Earnings in quarter of program entry 4,694 (3,453, 5,935) 459 
Earnings in 1st quarter after program entry 4,638 (3,195, 6,081) 459 
Earnings in 2nd quarter after program entry 6,057 (4,679, 7,435) 459 
Earnings in 3rd quarter after program entry 7,370 (5,983, 8,757) 459 
Earnings in 4th quarter after program entry 9,322 (7,707, 10,937) 459 
Earnings in 5th quarter after program entry 9,831 (8,168, 11,493) 459 
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Outcome | Mean 95% Confidence Interval Sample Size 
Earnings ($), among those not employed ina 
job related to training 
Earnings in 7th quarter before program entry 4,402 (3,797, 5,007) 632 
Earnings in 6th quarter before program entry 4,425 (3,826, 5,025) 793 
Earnings in 5th quarter before program entry 4,630 (4,056, 5,203) 793 
Earnings in 4th quarter before program entry 5,040 (4,409, 5,670) 793 
Earnings in 3rd quarter before program entry 5,142 (4,423, 5,861) 793 
Earnings in 2nd quarter before program entry 5,326 (4,618, 6,033) 793 
Earnings in 1st quarter before program entry 5,156 (4,340, 5,972) 793 
Earnings in quarter of program entry 4,953 (4,073, 5,832) 793 
Earnings in 1st quarter after program entry 5,289 (4,467, 6,110) 793 
Earnings in 2nd quarter after program entry 6,117 (5,237, 6,997) 793 
Earnings in 3rd quarter after program entry 6,970 (5,991, 7,949) 793 
Earnings in 4th quarter after program entry 7,337 (6,164, 8,510) 793 
Earnings in 5th quarter after program entry 8,080 (6,757, 9,403) 793 
Earnings of $6,240 or more in 5th quarter after 49.3 (41.4, 57.1) 2,355 
program entry (%) 
Cumulative earnings ($) in the 1st through 5th 30,108 (26,139, 34,077) 3.355 
quarters 
Change of Earnings from 3" quarter before 
program entry to 5 quarter after program 
entry 
Among those who completed program 2,224 (1,431, 3,018) 1,140 
Among those who left without completing 2,715 (1,860, 3,570) 431 
program 
Among those who still enrolled in program 1,422 (585, 2,260) 298 
Earnings in 5* quarter after program entry 
Among those who completed program 7,472 (6,033, 8,911) 1,140 
Among those who left without completing 7,264 (5,888, 8,639) 431 
program 
Among those who still enrolled in program 6,229 (5,284, 7,174) 298 
Employment Status (full sample) (%) 
Ever employed after finishing/leaving program 74.5 (68.2, 80.8) 2,211 
Currently employed after finishing/leaving 66.9 (61.3, 72.5) 2,211 
program 
Training-Related Employment (full sample) (°%) 
Ever employed after finishing/leaving program in 29.9 (21.4, 38.3) 2,211 
a job related to training 
Currently employed after finishing/leaving 27.3 (19.1, 35.5) 2,211 
program in a job related to training 
Ever employed after finishing classes in a job 6.1 (17.9, 34.4) 2,211 
related to training 
Currently employed after finishing classes in a job 24.0 (15.8, 32.1) 2,211 
related to training 
Current Employment (full sample) (%) 
Currently employed full-time 56.4 (50.6, 62.2) 2,211 
Currently employed part-time 10.4 (8.8, 12.1) 2,211 
Currently underemployed - employed part-time, 16 (6.5, 8.8) 2,211 
want to work full-time 
Currently employed part-time, do not want to 28 (1.9, 3.8) 2,211 
work full-time 
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Outcome | Mean 95% Confidence Interval Sample Size 

Current Employment Benefits (full sample) (%) 
Currently employed in job with health insurance 48.3 (41.6, 55.1) 2,211 
Currently employed in job with paid sick days 38.3 (31.2, 45.4) 2,211 
Currently employed in job with health insurance 35.3 (28.3, 42.4) 2,211 
and paid sick 
Currently employed in job with paid vacation 45.8 (38.7, 52.9) 2,211 
Currently employed in job with paid holidays 45.0 (37.8, 52.3) 2,211 
Currently employed in job with retirement or 42.3 (35.4, 49.3) 2,211 
pension benefits 

Reason for Wanting Part-time Work (full 

sample) (%) 
Child care problems 6.0 (-0.8, 12.8) 63 
Other family or personal reasons 51.7 (36.0, 67.4) 63 
Health or medical limitation 7.5 (-2.5, 17.6) 63 
Retired or Social Security 7.4 (-0.7, 15.5) 63 
Satisfied with income from part-time work 29.8 (14.9, 44.7) 63 

Employment Status (among those who finished 

classes) (%) 
Ever employed after finishing classes 90.3 (87.6, 93.0) 1,316 
Currently employed after finishing classes 81.8 (77.3, 86.4) 1,316 

Training-Related Employment (among those 

who finished classes) (%) 
Ever employed after finishing classes in a job 44.0 (32.9, 55.2) 1,316 
related to training 
Currently employed after finishing classes in a job 40.4 (28.8, 51.9) 1,316 
related to training 

Current Employment (among those who 

finished classes) (%) 
Currently employed full-time 71.0 (65.6, 76.5) 1,316 
Currently employed part-time 10.8 (8.3, 13.2) 1,316 
Currently underemployed - employed part-time, 79 (6.4, 9.5) 1,316 
want to work full-time 
Currently employed part-time, do not want to 28 (1.3, 4.4) 1,316 
work full-time 

Current Employment Benefits (among those 

who finished classes) (%o) 
Currently employed in job with health insurance 61.3 (52.6, 70.1) 1,316 
Currently employed in job with paid sick days 49.5 (40.0, 59.0) 1,316 
Currently employed in job with health insurance 46.3 (36.8, 55.7) 1,316 
and paid sick 
Currently employed in job with paid vacation 58.8 (49.4, 68.2) 1,316 
Currently employed in job with paid holidays 58.0 (48.5, 67.5) 1,316 
Currently employed in job with retirement or 541 (45.0, 63.2) 1,316 
pension benefits 

Reason for Wanting Part-time Work (among 

who finished classes) (%o) 
Child care problems 0 N/A 25 
Other family or personal reasons 54.8 (32.3, 77.4) 25 
Health or medical limitation 10.6 (-1.6, 22.8) 25 
Retired or Social Security 11.4 (-3.8, 26.7) 25 
Satisfied with income from part-time work 26.7 (4.6, 48.8) 25 
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Outcome | Mean 95% Confidence Interval Sample Size 

Employment Status (among those who left 

without completing) (%) 
Ever employed after leaving program 88.7 (86.4, 91.0) 519 
Currently employed after leaving program 77.7 (73.8, 81.6) 519 

Training-Related Employment (among those 

who left without completing) (%) 
Ever employed after leaving program in a job 15.9 (12.1, 19.7) 519 
related to training 
Currently employed after leaving program in a job 14.2 (10.9, 17.4) 519 
related to training 

Current Employment (among those who left 

without completing) (%) 
Currently employed full-time 60.5 (56.2, 64.9) 519 
Currently employed part-time 17.2 (14.2, 20.2) 519 
Currently underemployed - employed part-time, 12.4 (10.2, 14.5) 519 
want to work full-time 
Currently employed part-time, do not want to 48 (3.4, 6.6) 519 
work full-time 

Current Employment Benefits (among those 

who left without completing) (%) 
Currently employed in job with health insurance 50.6 (45.8, 55.4) 519 
Currently employed in job with paid sick days 37.9 (31.2, 44.5) 519 
Currently employed in job with health insurance 33.5 (27.4, 39.6) 519 
and paid sick 
Currently employed in job with paid vacation 46.1 (41.6, 50.7) 519 
Currently employed in job with paid holidays 45.1 (38.8, 51.3) 519 
Currently employed in job with retirement or 43.5 (37.2, 49.8) 519 
pension benefits 

Reason for Wanting Part-time Work (among 

those who left without completing) (%) 
Child care problems 10.0 (-0.3, 20.3) 38 
Other family or personal reasons 49.7 (31.7, 67.7) 38 
Health or medical limitation 5.5 (-8.8, 19.7) 38 
Retired or Social Security 4.7 (-4.2, 13.6) 38 
Satisfied with income from part-time work 31.9 (12.5, 51.4) 38 


G.2. EXPANDED RESULTS FOR CHAPTER 5 


Exhibits G.6 through G.9 contain expanded results of the analysis of outcomes by participant 
characteristics presented in Chapter 5. 


Exhibit G-6. Expanded Results for Program Completion by Participant Characteristics 


Mean Mean : Standard aoe 

Subgroup (Group 1) (Group 2) Difference Error eecoleuabeloe 

Male vs. Female (%) 50.1 58.8 -8.7 5.4 (-1.9, 19.4) 
Race/ ethnicity (%) 

White vs. Hispanic 48.0 52.7 -4.7 6.4 (-17.3, 8.0) 

White vs. Black 48.0 59.7 -11.7 9.0 (-29.4, 6.0) 

White vs. Other 48.0 45.1 2.9 4.9 (-6.6, 12.4) 
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95% 
Mean Mean i Standard : 
Subgroup Difference Confidence 
(Group 1) | (Group 2) Error Interval 
Veteran (%) 
Citizen and veteran vs. Not citizen 44.1 52.0 7.9 4.4 (-0.8, 16.6) 
or veteran 
Public assistance (%o) 
Currently receiving TRA vs. Not 58.2 51.2 7.0 6.2 (-19.1, 5.1) 
currently receiving TRA 
Currently receiving TANF vs. Not 7 
currently receiving TANF 64.8 51.1 13.7 11.7 (-36.6, 9.2) 
Currently receiving SNAP vs. Not 2 
currently receiving SNAP 54.6 50.9 3.7 5.3 (-14.2, 6.8) 
Education Attainment (%) 
Less than high school vs. More 7 
than high school 46.9 55.3 8.4 3.1 (2.4, 14.4) 
Employment (%) 
Age < 25 currently employed vs. 43.1 47.3 43 4.4 (-12.8, 4.3) 
Age <25 work in previous year : : ' as 
Age <25 currently employed vs. 43.1 40.3 28 55 (-8.1, 13.6) 
Age <25 no work in previous year 
Age <25 currently employed vs. 7 2 - 
Age 25-34 currently employed wee pe 8.6 — Cast t8) 
Age <25 currently employed vs. 7 2 | 
Age 25-34 work in previous year aan cad ie ne (26.3)-3:5) 
Age <25 currently employed vs. 
Age 25-34 no work in previous 43.1 57.7 -14.6 6.8 (-27.9, -1.3) 
year 
Age <25 currently employed vs. 2 7 _ 
Age 35+ currently employed 43.1 54.9 11.8 5.4 (-22.5, -1.1) 
Age <25 currently employed vs. 43.1 64.7 21.6 5.3 (32.0, 11:2) 
Age 35+ work in previous year 
Age <25 currently employed VS. 43.1 523 9.2 77 (-24.2, 5.8) 
Age 35+ no work in previous year 
Duration (%) 
Greater than one semester vs. One 41.4 59.9 -18.5 11.2 (-3.4, 40.4) 
semester or less 
Previous industry (%) 
Manufacturing vs. Natural 54.6 53.0 1.6 10.5 | (-19.1, 22.13) 
resources and mining 
Manufacturing vs. Construction 54.6 64.5 -10.0 6.2 (-22.3, 2.3) 
Manufacturing vs. Trade, 7 : 
transportation, utilities 54.6 55.0 0.5 5.6 (-11.6, 10.6) 
Manufacturing vs. Information, 
finance, professional, business 54.6 53.0 1.6 6.4 (-10.9, 14.1) 
services 
Manufacturing vs. education, 54.6 61.4 6.8 8.1 (-22.7, 9.1) 
health, government 
Manufacturing vs. Leisure and 54.6 46.2 8.4 8.2 (-7.8, 24.6) 
hospitality 
Manufacturing vs. Other services 54.6 54.4 0.2 7.9 (-15.4, 15.7) 
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Exhibit G-7. Expanded Results for Training-Related Employment by Participant Characteristic 


95% 
Mean Mean F Standard 
Subgroup Difference Confidence 
(cf colt] os i (CT Col!) 9) Error Interval 
Male vs. Female 32.4 35.6 -3.2 12.3 (-20.9, 27.3) 
Race/ ethnicity 
White vs. Hispanic 35.6 37.3 -1.8 5.1 (-11.8, 8.3) 
White vs. Black 35.6 22.1 13.5 4.8 (4.0, 22.9) 
White vs. Other 35.6 37.4 -1.8 5.1 (-11.8, 8.2) 
Veteran (%) 
Citizen and veteran vs. Not citizen 30.1 33.2 -3.1 4.5 (-5.6, 11.8) 
or veteran 
Public assistance (%) 
Currently receiving TRA vs. Not 35.4 32.9 2 8.9 | (-20.0, 14.9) 
currently receiving TRA 
Currently receiving TANF vs. Not : : 
currently receiving TANF 32.3 32.9 0.6 9.9 (-18.8, 20.0) 
Currently receiving SNAP vs. Not : 
currently receiving TANF 21.8 34.4 12.6 3.9 (5.0, 20.2) 
Education Attainment (%) 
Less than high school vs. More than 8.1 37.0 -8.9 4.0 (1.1, 16.7) 
high school 
Employment (%) 
Age < 25 currently employed vs. : 
Age <25 work in previous year on ope a = Oe. 
Age <25 currently employed vs. Age | 39g 20.2 12.7 4.7 (3.5, 21.8) 
<25 no work in previous year 
Age <25 currently employed vs. Age : 2 7 
25-34 currently employed 32.9 41.0 8.2 4.1 (-16.2, -0.1) 
Age <25 currently employed vs. Age 32.9 31.5 1.4 4.7 (-7.9, 10.7) 
25-34 work in previous year 
Age <25 currently employed vs. Age 32.9 33:6 9.3 6.4 (-3.3, 21.8) 
25-34 no work in previous year 
Age <25 currently employed vs. Age 32.9 35.5 27 53 (-13.0, 7.6) 
35+ currently employed 
BOS sre curently employed ve. NGe | —“a9G 33.0 0.2 5.7 | (11.3, 10.9) 
35+ work in previous year 
Age 25 CUIENEY EMPIOYEANSHAGE || 45.9 23.1 9.7 6.7 (-3.5, 23.0) 
35+ no work in previous year 
Duration (%) 
Greater than one semester vs. One 39.9 28.0 11.9 8.6 (-28.7, 5.0) 
semester or less 
Previous industry (%) 
Manufacturing vs. Natural resources 40.3 29.1 “111 8.4 (-5.4, 27.7) 
and mining 
Manufacturing vs. Construction 40.3 32.5 -7.7 7.4 (-6.9, 22.4) 
Manufacturing vs. Trade, 
transportation, utilities 40.3 27.7 12.6 5.1 (2.5, 22.7) 
Manufacturing vs. Information, 
finance, professional, business 40.3 24.6 -15.6 5.3 (5.2, 26.1) 
services 
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95% 
epee ( Beas Difference eee Confidence 
*) 1?) Interval 


Subgroup 


Manufacturing vs. education, health, 


government 40.3 32.4 -7.9 9.8 (-11.3, 27.1) 
Manufacturing vs. Leisure and 40.3 19.7 -20.6 5.3 (10.1, 31.1) 
hospitality 
Manufacturing vs. Other services 40.3 28.8 -11.5 7.3 (2.9, 25.8) 
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Exhibit G-8. Expanded Results for Change in Earnings by Participant Characteristic 


95% 
Mean Mean : Standard 5 
Subgroup Difference Confidence 
(Group 1) | (Group 2) Error Interval 
Male vs. Female ($) 2,246 2,923 -677 505 (-1667, 313) 
Veteran ($) 
Citizen and veteran vs. Not citizen 3,511 2,242 1,269 858 (-412, 2950) 
or veteran 
Public assistance ($) 
Cirrently irecelving: LE ve< Noy -2,218 2,443 -4,661 1,674 | (-7941, -380) 
currently receiving TRA 
Curmenty receiving TANF vs. Not 4,232 2,334 1,898 1,183 | (-421, 4217) 
currently receiving TANF 
Currently receiving SNAP vs. Not 1,511 2,471 -960 347 (-1640, -279) 
currently receiving SNAP 
Education Attainment ($) 
Less than high school vs. More : 7 
than high school 2,328 2,328 55 282 (-607, 497) 
Duration ($) 
Greater than one semester vs. One 3,358 1,680 1,678 500 (698, 2658) 
semester or less 
Employment ($) 
Bge < 25 culrently employed vs. 4,061 2,771 1,290 647 (22, 2559) 
Age <25 work in previous year 
Age <25 currently employed vs. 7 
Age <25 no work in previous year Ba 2/200 om 708 paet2t82) 
Age <25 currently employed vs. 
Age 25-34 currently employed 4,061 2,159 1,902 447 (1026, 2778) 
Age <25 currently employed vs. 4,061 952 3,109 656 | (1822, 4395) 
Age 25-34 work in previous year 
Age <25 currently employed vs. 4,061 3,042 1,019 904 | (-754, 2792) 
Age 25-34 no work in previous year 
Age <25 currently employed vs. 
Age'35+ cuiently employed 4,061 2,494 1,567 655 (283, 2851) 
Age <25 currently employed vs. 4,061 83 3,978 997 | (2024, 5931) 
Age 35+ work in previous year 
Age <25 currently employed vs. 4,061 2,038 2,023 860 | (337, 3708) 
Age 35+ no work in previous year 
Previous industry (%) 
Manufacturing vs. Natural 2,052 3920 -1867 1,478 | (-4764, 1029) 
resources and mining 
Manufacturing vs. Construction 2,052 1007 1046 964 (-844, 2936) 
Manufacturing vs. Trade, - . 
transportation, utilities 2,052 2194 142 741 (-1595, 1311) 
Manufacturing vs. Information, 
finance, professional, business 2,052 -266 2319 1,183 (0, 4638) 
services 
Manufacturing vs. education, 2,052 1143 910 970 (-991, 2810) 
health, government 
Manufacturing vs. Leisure and 2,052 1491 562 1,023 | (-1444, 2568) 
hospitality 
Manufacturing vs. Other services 2,052 2398 -345 1,319 (-2930, 2239) 
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Exhibit G-9. Expanded Results for Receipt of Public Assistance Benefits by Participant 
Characteristic 


95% 
Mean Mean A Standard 4 
Subgroup Difference Confidence 
(Group 1) | (Group 2) Error Titanual 
Male vs. Female 18.6 23.6 -5.0 3.7 (-2.3, 12.3) 
Race/ ethnicity 
White vs. Hispanic 16.2 17.6 -1.4 2.9 (-7.2, 4.3) 
White vs. Black 16.2 26.7 -10.5 2.1 (-14.5, -6.5) 
White vs. Other 16.2 22.5 -6.3 4.1 (-14.3, 1.7) 
Veteran (%) 
Citizen and veteran vs. Not citizen 31.3 18.2 13.1 3.4 (-19.8, -6.5) 
or veteran 
Public assistance (%o) 
CUO y Neca Ivinnd Ne; et 53.3 18.7 34.6 9.9 (-54.0, -15.2) 
currently receiving TRA 
Currently receiving TANF vs. Not : 
currently receiving TANF 65.1 18.6 46.6 10.4 (-67.1, -26.0) 
Currently receiving SNAP vs. Not " 7 
currently receiving SNAP 64.0 13.8 50.3 3.7 (-57.5, -43.0) 
Education Attainment (%) 
Less than high school vs. More 
than high school 20.2 18.6 1.6 1.8 (-5.1, 1.9) 
Employment (%) 
Age < 25 currently employed vs. , 7 
Age <25 work in previous year PLO a2 pe a2 Cit) 
Age <25 currently employed vs. 11.0 19.5 8.5 3.5 (-15.5, -1.5) 
Age <25 no work in previous year 
Age <25 currently employed vs. : ? 4 
Age 25-34 currently employed 11.0 15.4 4.4 2.1 (-8.5, -0.3) 
Age <25 currently employed vs. : 7 : 
Age 25-34 work in previous year ih 264 al a2 G2h6, 22:2) 
Age <25 currently employed vs. 
Age 25-34 no work in previous 11.0 28.9 -18.0 6.2 (-30.1, -5.8) 
year 
Age <25 currently employed vs. - ? 7 
Age 35+ currently employed 11.0 17.3 6.3 2.6 (-11.4, -1.2) 
Age <25 currently employed vs. 11.0 32.2 21.2 3.9 (-28.9, -13.5) 
Age 35+ work in previous year 
Age <25 currently employed vs. 11.0 40.3 -29,3 5.2 (-39.4, -19.2) 
Age 35+ no work in previous year 
Duration (%) 
Greater than one semester vs. One 18.6 20.0 1.4 2.8 (-4.2, 7.0) 
semester or less 
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95% 
( eet ( ae Difference eee Confidence 
P id Interval 


Subgroup 


Previous industry (%) 
Manufacturing vs. Natural 32.1 25.6 es 9.5 C121095.4) 
resources and mining 
Manufacturing vs. Construction 32.1 28.6 3.5 8.0 (-12.2, 19.2) 
Manufacturing vs. Trade, é 
transportation, utilities 32.1 23.2 8.9 6.7 (-4.3, 22.0) 
Manufacturing vs. Information, 
finance, professional, business 32.1 20.9 11.2 5.7 (0.1, 22.4) 
services 
Manufacturing vs. education, 32.1 29.5 2.7 7.3 (-11.8, 17.1) 
health, government 
Manufacturing vs. Leisure and 7 
hospitality 32.1 28.6 3.6 8.8 (-13.9, 21.0) 
Manufacturing vs. Other services 32.1 27.1 -5.0 8.3 (-11.2, 21.3) 


G.3. EXPANDED RESULTS FOR CHAPTER 6 


Exhibits G-10 through G-38 contain expanded results for the analysis of participant outcomes by 
service receipt reported in Chapter 6. 


Exhibit G-10. Expanded Results for the Analysis of Accelerated Learning Services on Program 
Completion in Chapter 6 


Expected Actual Excess 


Service Utilization Subgroup Completion Completion | Completion rato PoE: 
Rate (%) Rate (%) Rate 

Received Transfer Credits 

Yes 42.9 43.5 0.6 5.0 

No 53.6 53.4 -0.2 3.2 

Difference -10.7 “9.9 0.8 3.7 B35 
Received Credit for Prior Learning 

Yes 43.2 40.1 -3.1 5.0 

No 51.7 51.9 0.2 3.4 

Difference 85 -11.8 3.3 5.4 542 
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Exhibit G-11. Expanded Results for the Analysis of Persistence and Completion Services on 
Program Completion 
Expected PXoidtl-] | Excess Standard 
Error 


Service Utilization Subgroup Completion | Completion Completion 
Rate (%) Rate (%) Rate 


p-Value 


Received Career Counseling 


Yes 51.0 54.1 3.1 3.0 

No 51.5 49.9 -1.6 3.8 

Difference -0.5 4.2 4.7 25 058 
Received Academic Advising 

Yes 46.1 42.6 -3.5 3.4 

No 56.2 59.4 3.2 3.7 

Difference -10.1 -16.9 -6.8 3.1 027 
Received Financial Aid Advising 

Yes 45.9 43.4 -2.5 3.2 

No 53.9 55.1 1.2 3.6 

Difference -8.0 -11.6 3.6 2.3 122 


Took Any Course in Study Skills, 
Workplace Skills, or General Life 


Skills 
Yes 49.9 51.6 1.7 3.1 
No 52.6 51.2 -1.4 3.8 
Difference -2.7 0.4 3.1 22 .160 


Exhibit G-12. Expanded Results for the Analysis of Work-Based Learning on Program 
Completion 
Expected yXoi dt] | Excess a 
Error 


Service Utilization Subgroup Completion Completion | Completion 
Rate (%) Rate (%) Rate 


Fo EN 


Participated in Computer or Other 
Simulations to Practice Skills in a 
Virtual Setting 


Yes 47.3 48.7 1.4 4.6 
No 55.1 53.8 -1.2 3.6 
Difference -7,.7 -5.1 26 4.8 591 


Participated in Work-like Physical 
Environments 


Yes 52.7 56.6 3.9 3.2 
No 45.7 29.5 -16.2 3.8 
Difference 7.0 27.1 20.1 3S <.001 


Offered Opportunity for Direct 
Occupational Experience such as a 
Work Study Job or Internship 


Yes 52.3 55.9 3.6 4.0 
No 50.1 45.4 -4.7 4.2 
Difference 22 10.5 8.3 5.2 .109 
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Exhibit G-13. Expanded Results for the Analysis of Different Service Mixtures on Program 
Completion 
Expected PXo at] | Excess a 
ace) 


Service Mixture Completion | Completion Completion | Impact 
Rate (%) Rate (%) Rate 


oe 


Cluster 1. Mostly classroom 


instruction with few other 53.5 44.7 -8.9 -12.1 5.3 .022 
supports 

Cluster 2. Work-based 

learning with persistence and 44.6 40.2 -4.5 -6.2 2.8 .026 


completion services 
Cluster 3. Work-based 


learning with few other 51.5 52.2 0.8 1.0 3.1 .733 
supports 

Cluster 4. Work-based 

learning with employment- 58.6 77.5 19.0 23.2 2.3 <.001 


related services 


Exhibit G-14. Expanded Results for the Analysis of Accelerated Learning Services on Training- 
Related Employment (excluding participants still enrolled) 


Expected Actual Excess 
Employment Employment Employment 
Rate (%) Rate (%) Rate 


Service Utilization 
Subgroup 


Standard 
Error 


p-Value 


Transfer credits 

Yes 38.2 44.8 6.6 5.2 

No 31.7 30.3 -1.5 2.6 

Difference 6.5 14.5 8.0 4.0 .046 
Prior learning 

Yes 33.1 41.2 8.1 6.2 

No 32.9 32.6 -0.3 3.1 

Difference 0.2 8.6 8.4 6.6 .220 


Exhibit G-15. Expanded Results for the Analysis of Persistence and Completion Services on 
Training-Related Employment 


Expected Actual Excess 


Service Utilization Subgroup Employment Employment | Employment yen p-Value 
Rate (%) Rate (%) Rate 

Career counseling 

Yes 32.0 33.5 1.4 3.1 

No 33.4 32.6 -0.8 3.4 

Difference -1.4 0.9 2.2 29 441 
Academic advising 

Yes 31.8 31.3 -0.5 3.3 

No 33.8 34.2 0.4 3.0 

Difference -2.0 29 -0.9 2.1 666 
Financial aid advising 

Yes 34.1 38.2 4.2 3.5 

No 32.4 30.7 -1.7 2.8 

Difference 1.6 75 5.9 2.3 O13 
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Expected Actual Excess Standard 


ge) g 


Service Utilization Subgroup Employment Employment | Employment 
Rate (%) Rate (%) Rate 


p-Value 


Course in study skills, workplace 
skills, or general life skills 


Yes 32.4 34.0 1.6 3.3 
No 33.3 32.1 -1.3 3.2 
Difference -1.0 1.9 29 2.4 234 


Exhibit G-16. Expanded Results for the Analysis of Work-Based Learning on Training-Related 
Employment 


Expected Actual Excess 


Standard 


Service Utilization Subgroup Employment Employment Employment enor 


Rate (%) Rate (%) Rate 


p-Value 


Participated in computer or other 
simulations to practice skills in a 
virtual setting 


Yes 34.7 39.7 5.0 4.0 
No 31.4 27.2 -4.2 1.9 
Difference 3.3 12.5 9.2 31 003 


Participated in work-like physical 
environments 


Yes 33.4 36.2 2.8 3.1 
No 30.8 18.3 -12.5 2.5 
Difference 2.6 17.9 15.3 3.4 <.001 


Offered opportunity for direct 
occupational experience such as 
a work study job or internship 


Yes 34.6 41.0 6.4 3.8 
No 30.8 22.5 -8.3 2.1 
Difference 3.8 18.5 14,7 4.1 <.001 


Exhibit G-17. Expanded Results for the Analysis of Employment-Related Services on Training- 
Related Employment 


Expected Actual Excess 
Employment Employment Employment 
Rate (%) Rate (%) Rate 


Service Utilization Standard 


Error p-Nalue 


Subgroup 


Received job search or 
placement assistance 


Yes 33.3 43.8 10.4 3.7 
No 32.6 25.5 -7.2 2.3 
Difference 0.7 18.3 17.6 3.4 <.001 


Received interviewing 
practice services 


Yes 33.8 46.2 12.4 4.2 
No 32.6 27.9 -4.7 2.4 
Difference 1.2 18.3 17.1 3.6 <.001 
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Exhibit G-18. Expanded Results for the Analysis of Different Service Mixtures on Training- 


Related Employment 


Service Mixture 


Cluster 1. Mostly classroom 
instruction with few other 
supports 


Expected 
Employment Employment Employment Impact 
Rate (%) 


r Your] | 


Rate (%) 


22.1 


Excess 


Rate 


-13.0 


Standard 
ace) 


3.2 


p- 


<.001 


Cluster 2. Work-based 
learning with persistence 
and completion services 
Cluster 3. Work-based 
learning with few other 
supports 


34.4 


37.1 


33.8 


4.1 


5.5 


3.2 


2.7 


.086 


.760 


Cluster 4. Work-based 
learning with employment- 
related services 


41.3 


8.6 


10.8 


3.1 


001 


Exhibit G-19. Expanded Results for the Analysis of Accelerated Learning Services on Change in 


Earnings (all participants) 


Service Utilization Subgroup 


Received transfer credits 


Expected 
Change in 
Earnings 


169) 


rYoutt-] | 


Change in 
Earnings 


(63) 


Excess 


Change in 
Earnings 


(69) 


Standard 


Error 


p-Value 


Yes $2,622 $3,109 $487 $355 

No $2,030 $1,768 $-261 $373 

Difference $592 $1,341 $749 $378 -O51 
Received credit for prior 
learning or work experience 

Yes $2,267 $3,048 $781 $695 

No $2,235 $2,199 $-36 $332 

Difference $32 $849 $817 $656 .213 


Exhibit G-20. Expanded Results for the Analysis of Accelerated Learning Services on Change in 


Earnings (participants not still enrolled) 


Service Utilization Subgroup 


Received transfer credits 


Expected 
Change in 
Earnings 


169) 


rYoutt-] | 


Change in 
Earnings 


169) 


Excess 


Change in 
Earnings 


(69) 


Standard 


Error 


p-Value 


Yes $2,434 $3,538 $1,104 $449 
No $1,937 $1,815 $-122 $415 
Difference $496 $1,723 $1,226 $434 -005 
Received credit for prior 
learning or work experience 
Yes $2,054 $2,983 $930 $865 
No $2,096 $2,334 $239 $398 
Difference $-42 $649 $691 $763 365 
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Exhibit G-21. Expanded Results for the Analysis of Persistence and Completion Services on 
Change in Earnings (all participants) 


Expected Actual Excess 
Changein Changein Changein Standard 


Service Utilization Subgroup p-Value 


Earnings Earnings Earnings Error 


169) 169) (69) 


Received career counseling 

Yes $2,360 $2,100 $-260 $467 

No $2,170 $2,310 $140 $307 

Difference $191 $-210 $-400 $348 -250 
Received academic advising 

Yes $2,499 $2,261 $-238 $349 

No $2,003 $2,214 $212 $361 

Difference $496 $47 $-449 $256 .079 
Received financial aid advising 

Yes $2,590 $2,426 $-164 $388 

No $2,075 $2,150 $75 $379 

Difference $515 $276 $-239 $400 .550 
Took any course in study skills, 
workplace skills, or general life skills 

Yes $2,334 $2,322 $-12 $403 

No $2,1545 $2,164 $10 $334 

Difference $179 $158 $-22 $322 947 


Exhibit G-22. Expanded Results for the Analysis of Persistence and Completion Services on 
Change in Earnings (participants not still enrolled) 


Expected Actual Excess 
Changein Changein Changein Standard 


Service Utilization Subgroup p-Value 


Earnings Earnings Earnings Error 


169) 169) (69) 


Received career counseling 

Yes $2,173 $2,223 $50 $523 

No $2,051 $2,434 $383 $373 

Difference $122 $-210 $-333 $342 331 

Received academic advising 

Yes $2,318 $2,577 $259 $465 

No $1,920 $2,189 $269 $398 

Difference $398 $388 $-10 $309 .974 
Received financial aid advising 

Yes $2,333 $2,845 $512 $435 

No $1,996 $2,159 $163 $428 

Difference $337 $686 $349 $384 364 
Took any course in study skills, 
workplace skills, or general life skills 

Yes $2,142 $2,437 $294 $507 

No $2,055 $2,297 $241 $373 

Difference $87 $140 $53 $356 .881 
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Exhibit G-23. Expanded Results for the Analysis of Work-Based Learning of Change in Earnings 


(all participants) 


Service Utilization Subgroup 


Expected 
Change in 
Earnings 


Actual 
Change in 
Earnings 


Excess 
Change in 
Earnings 


Standard 


Error 


PoE 


169) 


169) 


(63) 


Participated in computer or other 
simulations to practice skills in a 
virtual setting 
Yes $2,254 $2,605 $352 $286 
No $2,221 $1,909 $-313 $391 
Difference $32 $697 $665 $321 -039 
Participated in work-like physical 
environments 
Yes $2,245 $2,127 $-118 $350 
No $2,199 $2,740 $541 $508 
Difference $45 $-613 $-659 $507 .194 
Offered opportunity for direct 
occupational experience such as a 
work study job or internship 
Yes $2,252 $2,212 $-40 $341 
No $2,214 $2,273 $60 $489 
Difference $38 $-61 $-99 $479 B35 


Exhibit G-24. Expanded Results for the Analysis of Work-Based Learning of Change in Earnings 


(participants not still enrolled) 


Service Utilization Subgroup 


Expected 
Change in 
Earnings 


69) 


Actual 
Change in 
Earnings 


(69) 


Excess 
Change in 
Earnings 


(69) 


Standard 
ae) g 


p-Value 


Participated in computer or 
other simulations to practice 
skills in a virtual setting 
Yes $2,068 $2,850 $782 $319 
No $2,116 $1,951 $-165 $456 
Difference $-48 $900 $947 $346 .006 
Participated in work-like 
physical environments 
Yes $2,100 $2,234 $134 $419 
No $2,064 $2,971 $908 $605 
Difference $36 $-737 $-774 $582 .184 
Offered opportunity for direct 
occupational experience such as 
a work study job or internship 
Yes $2,087 $2,329 $242 $405 
No $2,104 $2,403 $299 $570 
Difference $-17 $-74 $-57 $531 -914 
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Exhibit G-25. Expanded Results for the Analysis of Employment-related Services on Change in 
Earnings (all participants) 


Expected Actual Excess 
Change in Change in Change in 
Earnings ($) Earnings($) Earnings ($) 


Standard 
ane) g 


Service Utilization p-Value 


Subgroup 


Received job search or 
placement assistance 


Yes $2,242 $2,360 $118 $500 
No $2,232 $2,154 $-79 $312 
Difference $9 $206 $196 $459 669 


Received interviewing 
practice services 


Yes $2,115 $2,738 $623 $581 
No $2,279 $2,062 $-217 $272 
Difference $-163 $677 $840 $459 067 


Exhibit G-26. Expanded Results for the Analysis of Employment-related Services on Change in 

Earnings (participants not still enrolled) 

Expected Actual Excess Standard 
Error 


Service Utilization 
Subgroup 


Change in Change in Change in 
Earnings ($) Earnings($) Earnings ($) 


p-Value 


Received job search or 
placement assistance 


Yes $2,097 $2,564 $467 $559 
No $2,092 $2,215 $123 $381 
Difference $5 $350 $344 $483 -476 


Received interviewing 
practice services 


Yes $2,039 $2,887 $848 $632 
No $2,116 $2,152 $36 $344 
Difference $-77 $735 $812 $475 .088 


Exhibit G-27. Expanded Results for the Analysis of Service Mixtures on Change in Earnings (all 
participants) 


Expected Actual Excess 
Change Change Change 


Standard 


Service Mixture in in in Impact Error 


Earnings Earnings’ Earnings 


(69) (69) 169) 


p-Value 


Cluster 1. Mostly classroom 


instruction with few other $2,141 $2,199 $58 $78 $462 .865 
supports 

Cluster 2. Work-based learning 

with persistence and completion $2,640 $2,488 $-152 $-215 $454 .635 
services 


Cluster 3. Work-based learning 


with few other supports $1,994 $1,813 $-180 $-246 $273 368 
Cluster 4. Work-based learning 
with employment-related $2,079 $2,511 $432 $526 $461 .253 
services 
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Exhibit G-28. Expanded Results for the Analysis of Different Service Mixtures on Change in 
Earnings (participants not still enrolled) 


Expected Actual Excess 
Change Change Change 


Standard 


Service Mixture Th) in in Impact Error 


Earnings Earnings’ Earnings 


(63) (69) 169) 


p-Value 


Cluster 1. Mostly classroom 
instruction with few other $2,045 $2,154 $109 $-213 $449 .635 
supports 

Cluster 2. Work-based learning 
with persistence and completion $2,363 $2,879 $516 $339 $377 .369 
services 

Cluster 3. Work-based learning 
with few other supports 
Cluster 4. Work-based learning 
with employment-related $2,046 $2,664 $618 $442 $456 332 
services 


$1,920 $1,834 $-85 $-480 $321 .136 


Exhibit G-29. Expanded Results for the Analysis of Accelerated Learning Services on public 
assistance receipt 


Excess 
Public Standard 
Assistance Error 
Receipt (%) 


Expected 
Public 
Assistance 
Receipt (%) 


Actual 
Participation 
Rate (%) 


Service Utilization Subgroup 


fo ELC: 


Received transfer credits 


Yes 17.6 19.5 1.9 1.8 
No 19.9 19.4 -0.5 1.0 
Difference 2.3 0.1 2.4 21 266 


Received credit for prior 
learning or work experience 


Yes 21.4 31.7 10.3 4.7 
No 19.4 18.9 -0.5 0.9 
Difference 2.0 12.8 10.7 4.7 038 


Exhibit G-30. Expanded Results for the Analysis of Persistence and Completion Services on 
Public Assistance Receipt 


Expected Actual Public Bene Standard 
Service Utilization Subgroup Participation Assistance ‘Ascletance EOF p-Value 
Rate (%) Receipt (%) Receipt (%) 
Received career counseling 
Yes 21.9 25.5 3.6 1.3 
Did not 18.1 16.2 -1.9 1.0 
Difference 3.8 9.3 5:5) 1.6 <.001 
Received academic advising 
Yes 19.0 22.4 3.4 1.0 
No 19.8 16.7 -3.1 1.1 
Difference -0.8 5.7 6.5 1.2 <.001 
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Excess 
Public Standard 
Assistance Error 
Li C=Yot=} [9] a Cs) 


Expected Actual Public 


Service Utilization Subgroup Participation Assistance 
Rate (%) Receipt (%) 


Fo EN 


Received financial aid advising 


Yes 20.6 24.4 3.8 1.3 
No 18.9 17.1 -1.8 1.1 
Difference 1.6 7.3 5.6 1.8 002 


Took any course in study skills, 
workplace skills, or general life 


skills 
Yes 20.1 21.4 1.3 1.2 
No 18.9 17.8 -1.1 1.1 
Difference Ld1 3S 24 1.6 126 


Exhibit G-31. Expanded Results for the Analysis of Work-based Learning on Public Assistance 
Receipt 


Expected 


Public Actual Public Excess Public 


Standard 


Service Utilization Subgroup Assistance Assistance p-Value 


Assistance Error 


i ie) i ie) 
Receipt (%) Receipt (%) Receipt (%) 


Participated in computer or 
other simulations to practice 
skills in a virtual setting 


Yes 18.8 20.7 1.9 1.4 
No 20.1 18.3 -1.8 1.2 
Difference -1,3 24 3.7 2.0 O78 


Participated in work-like 
physical environments 


Yes 19.4 19.5 0.1 0.9 
No 19.8 19.3 -0.5 1.8 
Difference -0.5 0.1 0.6 1.8 739 


Offered opportunity for direct 
occupational experience such 
as a work study job or 


internship 
Yes 19.5 21.3 1.8 1.1 
No 19.4 17.0 -2.4 1.1 
Difference 0.1 4.2 4.2 14 .003 
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Exhibit G-32. Expanded Results for the Analysis of Employment-related Services on Public 
Assistance Receipt 


Expected Public Actual Public Excess Public 
: : A Standard 
Assistance Assistance Assistance paae p-Value 
Receipt (%) Receipt (%) Receipt (%) 


Service Utilization 


Subgroup 


Received job search or 
placement assistance 


Yes 20.5 21.4 0.9 1.2 
No 18.7 18.2 -0.6 1.2 
Difference 1.8 3.2 1.4 1.6 75 


Received interviewing 
practice services 


Yes 22.1 22.8 0.6 1.2 
No 18.5 18.3 -0.2 1.1 
Difference 3.6 4.4 0.9 1.6 585 


Exhibit G-33. Expanded Results for the Analysis of Different Service Mixtures on Public 
Assistance Receipt 


Expected 
Public 
Assistance 
Receipt (%) 


Actual Public Excess Public 
Assistance Assistance Impact 
Receipt (%) Receipt (%) 


Standard p- 
Error Value 


Service Mixture 


Cluster 1. Mostly 
classroom instruction 19.5 15.6 -3.9 -5.3 1.9 .007 
with few other supports 
Cluster 2. Work-based 
learning with persistence 20.0 25.6 5.7 7.9 2.1 <.001 
and completion services 
Cluster 3. Work-based 
learning with few other 17.7 16.3 -1.4 -1.9 1.6 .245 
supports 

Cluster 4. Work-based 
learning with 
employment-related 
services 


21.2 20.0 -1.2 -1.4 2.1 .489 


Exhibit G-34. Expanded Results for the Analysis of Accelerated Learning Services on Poverty 


Excess 

Poverty Standard 
Rate ane) g 
(9) 


Expected yXoi dt] | 
Service Utilization Subgroup Poverty Rate Poverty Rate 
(C/:)) (Cy) 


p-Value 


Received transfer credits 


Yes 30.1 28.5 -1.6 1.7 
No 35.5 36.0 0.4 1.0 
Difference -5.4 -7.4 -2.0 1.8 .273 


Received credit for prior 
learning or work experience 


Yes 30.9 32.1 1.2 4.0 
No 34.6 34.5 -0.1 0.9 
Difference -3.7 -2.4 1.3 3.8 744 
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Exhibit G-35. Expanded Results for the Analysis of Persistence and Completion on Poverty 


Excess 
Poverty Standard 


Expected Actual 
Service Utilization Subgroup Poverty Poverty 
Rate (%) Rate (%) 


Rate Error p-Value 


(€/9) 


Received career counseling 

Yes 37.0 35.6 -1.4 1.7 

No 33.1 33.8 0.8 1.1 

Difference 3.9 1.8 -2.2 2.1 301 
Received academic advising 

Yes 34.5 33.8 -0.6 1.3 

No 34.4 35.0 0.6 1.4 

Difference 0.1 -1.2 -1.2 1.9 .525 
Received financial aid advising 

Yes 36.4 36.6 0.2 1.7 

No 33.5 33.4 -0.1 1.1 

Difference 2.9 3.2 0.3 2.0 .876 
Took any course in study skills, 
workplace skills, or general life skills 

Yes 37.0 37.3 0.3 1.4 

No 32.3 32.0 -0.3 1.2 

Difference 4.7 5.3 0.6 1.8 -746 


Exhibit G-36. Expanded Results for the Analysis of Work-based Learning on Poverty 


Expected P Xo ate) | Excess 
Service Utilization Subgroup Poverty Rate | Poverty Rate | Poverty 
(CZ) (CZ) Rate (%) 


Standar 
d Error 


p-Value 


Participated in computer or other 
simulations to practice skills in a 
virtual setting 


Yes 31.5 31.1 -0.4 1.6 
No 37.1 37.5 0.3 1.3 
Difference -5.7 -6.4 -0.7 2.2 .750 


Participated in work-like physical 
environments 


Yes 34.5 34.2 -0.3 1.1 
No 34.0 35.2 1.2 2.3 
Difference 0.6 -0.9 -1.5 2.7 585 


Offered opportunity for direct 
occupational experience such as a 
work study job or internship 


Yes 34.8 34.4 -0.4 1.3 
No 34.0 34.4 0.5 1.3 
Difference 0.8 0.0 -0.9 1.9 .656 
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Exhibit G-37. Expanded Results for the Analysis of Employment-related Services on Poverty 


Expected Actual Excess 


Poverty Rate Poverty Rate Poverty Rate Standard 


Error 


Service Utilization p-Value 


Subgroup (C7) (C79) () 


Received job search or 
placement assistance 


Yes 36.6 35.4 -1.2 1.6 
No 33.0 33.8 0.8 1.0 
Difference 3.6 1.6 -2.0 1.7 .247 


Received interviewing 
practice services 


Yes 38.5 35.9 -2.7 2.0 
No 33.0 33.9 0.9 1.0 
Difference 5.5 1.9 -3.6 2.2 .103 


Exhibit G-38. Expanded Results for the Analysis of Different Service Mixtures on Poverty 


Excess 

Poverty Standard 
Rate Impact ane) g 
(9) 


Expected Actual 
Service Mixture Poverty Poverty 
Rate (%) Rate (%) 


fo E 


Cluster 1. Mostly classroom 
instruction with few other supports ane aca — cai ‘iis 
Cluster 2. Work-based learning with 353 35.1 0.2 0.3 22 892 
persistence and completion services 
Cluster 3. Work-based learning with 30.9 30.3 0.6 -0.8 2.0 706 
few other supports 
Cluster 4. Work-based learning with 38.1 37.7 0.4 -0.5 241 810 
employment-related services 
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G.4. EXPANDED RESULTS FOR CHAPTER 7 


Exhibits G-39 through G-42 contain expanded results of the analysis of outcomes by program reported 
in Chapter 7. 


Exhibit G-39. Expanded Results for the Analysis of Program Completion by Program (%) 


Program Mean 80 Percent 99.9 Percent 
Credible Interval Credible Interval 

CDL and Forklift (Cincinnati) 93.1 (89.8, 96.0) (82.2, 98.7) 
Core Plus (Delgado) 82.8 (79.3, 86.3) (72.7, 90.2) 
Licensed Practical Nurse (Washburn) 79.5 (73.3, 85.4) (61.6, 92.2) 
Mechanical Craft (Chaffey) 79.2 (69.2, 88.3) (49.7, 95.6) 
Emergency Medical Technician (Washburn) 74.5 (68.2, 80.5) (58.1, 87.8) 
Welding Technology (Washburn) 73.9 (63.5, 83.5) (45.6, 92.4) 
Advanced Manufacturing and Engineering (Chaffey) 67.7 (62.2, 73.1) (53.6, 80.4) 
Advanced Manufacturing (Manchester) 67.6 (63.8, 71.3) (57.5, 76.6) 
Welding Technology (Manchester) 66.1 (53.9, 78.0) (33.3, 91.4) 
Industrial Automation (Chaffey) 65.3 (54.9, 75.2) (39.2, 87.5) 
Industrial Maintenance (Chaffey) 61.2 (56.0, 66.2) (47.7, 73.4) 
TECH 101 (Delgado) 60.1 (55.6, 64.6) (48.1, 71.3) 
TRAMCON Advanced (Miami Dade) 56.6 (39.6, 73.1) (17.7, 88.9) 
TRAMCON Basic (Miami Dade) 54.1 (38.4, 69.8) (17.0, 87.1) 
Pre-engineering (Chaffey) 52.1 (37.2, 67.0) (17.2, 86.4) 
TRAMCON Foundation (Miami Dade) 51.1 (46.6, 55.6) (40.0, 62.4) 
Welding (South Central) 47.9 (41.0, 55.0) (31.2, 65.3) 
Advanced Welding Bootcamp and Program (Bossier) 42.7 (31.8, 53.8) (18.7, 70.6) 
Fast Track to Manufacturing (Bossier) 42.5 (29.1, 56.1) (14.3, 74.7) 
Certified Production Technician (South Central) 32.8 (25.5, 40.2) (15.4, 52.6) 
Right Skills Now (South Central) 27.4 (13.1, 43.5) (2.9, 70.9) 
HVAC (Chaffey) 27.3 (20.2, 34.7) (10.7, 47.3) 
Gateway Courses to IT Programs (Ivy Tech) 19.2 (13.0, 25.8) (6.1, 37.6) 
Machining (South Central) 16.6 (8.7, 25.6) (2.3, 43.0) 
Informatics (Ivy Tech) 13.7 (6.7, 21.9) (1.7, 40.0) 
Welding (Chaffey) 12.0 (7.0, 17.5) (2.6, 30.1) 
Mechatronics (South Central) 10.7 (4.9, 17.5) (1.3, 33.0) 
Network Infrastructure (Ivy Tech) 9.8 (4.6, 15.9) (1.3, 29.5) 
Software Development (Ivy Tech) 8.6 (4.7, 12.9) (1.6, 22.5) 
Information Technology Support (Ivy Tech) 4.2 (1.7, 7.2) (0.3, 15.8) 
Cyber Security (Ivy Tech) 4.1 (1.8, 6.9) (0.4, 14.3) 
Database Management (Ivy Tech) 3.7 (0.6, 8.1) (0, 29.3) 

Computer Science (Ivy Tech) 3.1 (0.7, 6.5) (0.1, 18.3) 
Server Administration (Ivy Tech) 2.5 (0.5, 5.3) (0, 16.3) 
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Exhibit G-40. Expanded Results for the Analysis of Finding Training-related Employment by 
Program (%) (Subset to participants not still enrolled at survey follow-up) 


Program Mean 80 Percent 99.9 Percent 
Credible Interval Credible Interval 
Licensed Practical Nurse (Washburn) 83.8 (78.1, 89.1) (66.4, 94.9) 
Welding Technology (Washburn) 56.7 (45.5, 68.0) (27.8, 81.2) 
Advanced Manufacturing (Manchester) 56.0 (52.1, 60.0) (45.9, 66.8) 
Mechanical Craft (Chaffey) 54.4 (42.6, 66.4) (26.1, 81.2) 
Machining (South Central) 52.0 (35.4, 68.7) (15.4, 89.1) 
Pre-engineering (Chaffey) 45.7 (31.3, 60.5) (14.2, 80.1) 
Industrial Automation (Chaffey) 44.9 (34.6, 55.5) (21.2, 72.0) 
Advanced Manufacturing and Engineering (Chaffey) 41.3 (35.5, 47.2) (26.9, 55.9) 
Certified Production Technician (South Central) 39.8 (31.5, 48.4) (20.7, 63.6) 
Advanced Welding Bootcamp and Program (Bossier) 37.5 (27.3, 48.1) (14.6, 65.2) 
Welding (South Central) 37.2 (30.0, 44.7) (19.2, 56.0) 
Welding Technology (Manchester) 35.2 (23.2, 47.7) (9.8, 67.7) 
Industrial Maintenance (Chaffey) 33.3 (28.4, 38.4) (21.2, 45.8) 
Fast Track to Manufacturing (Bossier) 32.2 (20.4, 44.7) (8.5, 64.0) 
TRAMCON Basic (Miami Dade) 28.2 (14.7, 42.6) (3.7, 68.3) 
Right Skills Now (South Central) 27.6 (12.1, 45.4) (1.8, 78.1) 
CDL and Forklift (Cincinnati) 27.5 (21.7, 33.4) (14.2, 43.3) 
Mechatronics (South Central) 27.5 (13.7, 43.1) (3.8, 69.1) 
TRAMCON Foundation (Miami Dade) 25.7 (21.8, 29.7) (16.5, 36.2) 
Emergency Medical Technician (Washburn) 24.2 (18.5, 30.2) (11.6, 40.4) 
HVAC (Chaffey) 23.2 (15.0, 32.0) (6.9, 47.2) 
TECH 101 (Delgado) 22.9 (18.9, 27.1) (13.8, 33.4) 
Welding (Chaffey) 20.2 (12.8, 28.2) (5.7, 43.6) 
Gateway Courses to IT Programs (Ivy Tech) 19.1 (12.6, 26.0) (6.1, 38.6) 
TRAMCON Advanced (Miami Dade) 17.6 (7.2, 29.8) (1.3, 57.4) 
Core Plus (Delgado) 17.5 (14.0, 21.0) (9.7, 28.2) 
Network Infrastructure (Ivy Tech) 12.5 (5.8, 20.4) (1.4, 37.4) 
Server Administration (Ivy Tech) 12.3 (4.2, 22.4) (0.5, 48.8) 
Cyber Security (Ivy Tech) 12.2 (6.3, 18.9) (2.0, 34.4) 
Informatics (Ivy Tech) 7.3 (2.1, 14.1) (0.2, 34.9) 
Software Development (Ivy Tech) 7.3 (3.4, 11.8) (0.9, 23.3) 
Database Management (Ivy Tech) 7.2 (1.2, 15.7) (0.1, 59.3) 
Information Technology Support (Ivy Tech) 6.7 (2.8, 11.2) (0.4, 24.0) 
Computer Science (Ivy Tech) 5.5 (1.3, 11.1) (0.1, 29.2) 
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Exhibit G-41. Expanded Results for the Analysis of Change in Earnings by Program ($) (Subset 
to participants not still enrolled at survey follow-up) 


Program 


Server Administration (Ivy Tech) 
Cyber Security (Ivy Tech) 


Mean 


5066 
4835 


80 Percent 


Credible Interval 


(3640, 6694) 
(3347, 6530) 


99.9 Percent 


Credible Interval 


(1775, 9880) 
(1259, 8963) 


Mechatronics (South Central - Ridgewater and SCCC) 


4380 


(3414, 5404) 


(1765, 7549) 


Computer Science (Ivy Tech) 


4222 


(2929, 5562) 


(665, 8930) 


Network Infrastructure (Ivy Tech) 


3753 


(2626, 5074) 


(1104, 7833) 


Informatics (Ivy Tech) 


3697 


(2657, 4728) 


(920, 6576) 


Machining (South Central) 


3612 


(2588, 4739) 


(1004, 6517) 


Welding Technology (Washburn) 


3420 


(2610, 4288) 


(1523, 5745) 


Licensed Practical Nurse (Washburn) 


3239 


(2709, 3779) 


(1945, 4680) 


Database Management (Ivy Tech) 
Software Development (Ivy Tech) 


3112 
3040 


(1438, 4810) 
(2430, 3675) 


(-1029, 7844) 
(1467, 4822) 


Welding (South Central) 


2998 


(2220, 3837) 


(1109, 5209) 


Welding Technology (Manchester) 


2907 


(2145, 3701) 


(744, 5117) 


Information Technology Support (Ivy Tech) 


2561 


(1815, 3354) 


(692, 4915) 


Mechanical Craft (Chaffey) 


2522 


(1806, 3268) 


(730, 4501) 


Right Skills Now (South Central) 


2404 


(1358, 3424) 


(-766, 5325) 


Advanced Manufacturing (Manchester) 
Welding (Chaffey) 


2304 
2201 


(1870, 2748) 
(1581, 2824) 


(1225, 3458) 
(643, 3930) 


Certified Production Technician (South Central) 
Gateway Courses to IT Programs (Ivy Tech) 


2088 
1935 


(1437, 2786) 
(1145, 2749) 


(426, 4114) 
(-335, 4530) 


Emergency Medical Technician (Washburn) 


1835 


(1363, 2296) 


(643, 3027) 


Advanced Manufacturing and Engineering (Chaffey) 


1667 


(1159, 2186) 


(394, 3028) 


HVAC (Chaffey) 


1544 


(874, 2211) 


(-242, 3281) 


TRAMCON Advanced (Miami Dade) 


1476 


(613, 2307) 


(-894, 3831) 


Advanced Welding Bootcamp and Program (Bossier) 


1467 


(808, 2129) 


(-308, 3418) 


Fast Track to Manufacturing (Bossier) 
Industrial Automation (Chaffey) 


1327 
1270 


(563, 2124) 
(453, 2037) 


(-711, 3541) 
(-1152, 3432) 


TRAMCON Basic (Miami Dade) 


1088 


(252, 1950) 


(-1182, 3767) 


Industrial Maintenance (Chaffey) 


1074 


(693, 1460) 


(156, 2069) 


Pre-engineering (Chaffey) 


1011 


(216, 1796) 


(-1397, 3203) 


TECH 101 (Delgado) 


856 


(520, 1189) 


(-3, 1681) 


TRAMCON Foundation (Miami Dade) 


683 


(361, 1000) 


(-172, 1476) 


CDL and Forklift (Cincinnati) 
Core Plus (Delgado) 


665 
422 


(261, 1077) 
(113, 733) 


(-352, 1673) 
(-337, 1173) 
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Exhibit G-42. Expanded Results for the Analysis of Public Assistance Receipt by Program (%) 


Program ES Ceabie ee ee acieee 
Welding (Chaffey) 38.0 (33.8, 42.6) (26.9, 51.9) 
Core Plus (Delgado) 26.5 (23.8, 29.3) (19.3, 35.0) 
Industrial Automation (Chaffey) 25.5 (21.7, 29.3) (13.6, 37.8) 
Pre-engineering (Chaffey) 24.8 (20.8, 28.8) (13.6, 40.6) 
Cyber Security (Ivy Tech) 23.5 (19.6, 25.9) (14.1, 34.3) 
CDL and Forklift (Cincinnati) 22.6 (19.2, 25.9) (13.1, 31.3) 
HVAC (Chaffey) 21.8 (18.7, 25.2) (13.4, 33.0) 
TRAMCON Foundation (Miami Dade) 21.4 (19.0, 24.0) (16.1, 29.2) 
Right Skills Now (South Central) 21.2 (16.8, 25.8) (9.3, 37.3) 
Fast Track to Manufacturing (Bossier) 21.2 (17.1, 26.0) (11.7, 42.4) 
Computer Science (Ivy Tech) 21.1 (16.5, 25.7) (10.1, 34.9) 
Advanced Manufacturing (Manchester) 20.8 (18.6, 23.0) (15.2, 27.5) 
TECH 101 (Delgado) 20.4 (18.0, 22.8) (14.2, 27.8) 
Server Administration (Ivy Tech) 18.2 (14.1, 22.5) (7.9, 33.1) 
Network Infrastructure (Ivy Tech) 17.6 (14.3, 20.9) (7.6, 28.2) 
Industrial Maintenance (Chaffey) 17.2 (14.6, 19.7) (10.0, 24.3) 
TRAMCON Advanced (Miami Dade) 17.0 (13.7, 20.4) (8.0, 30.5) 
TRAMCON Basic (Miami Dade) 16.9 (13.7, 20.3) (7.7, 30.8) 
Welding Technology (Manchester) 16.8 (13.7, 19.9) (8.6, 29.4) 
Licensed Practical Nurse (Washburn) 16.5 (13.3, 19.9) (9.3, 27.6) 
Advanced Welding Bootcamp and Program (Bossier) 16.4 (13.5, 19.6) (9.2, 29.4) 
Software Development (Ivy Tech) 16.2 (13.5, 19.1) (8.9, 26.2) 
Database Management (Ivy Tech) 16.0 (11.4, 20.9) (5.0, 33.2) 
Information Technology Support (Ivy Tech) 15.9 (13.2, 18.5) (8.2, 24.0) 
Mechatronics (South Central) 15.8 (12.3, 19.7) (7.9, 29.7) 
Welding (South Central) 15.3 (13.0, 18.0) (9.6, 24.3) 
Certified Production Technician (South Central) 15.2 (12.7, 17.7) (7.6, 23.6) 
Machining (South Central) 14.8 (11.9, 17.9) (7.4, 28.6) 
Informatics (Ivy Tech) 14.6 (10.9, 18.2) (4.9, 25.5) 
Gateway Courses to IT Programs (Ivy Tech) 14.4 (11.7, 17.7) (8.6, 29.1) 
Welding Technology (Washburn) 12.9 (10.1, 15.6) (5.0, 21.5) 
Advanced Manufacturing and Engineering (Chaffey) 12.3 (9.8, 14.6) (5.9, 18.6) 
Mechanical Craft (Chaffey) 9.4 (6.7, 12.0) (2.7, 18.0) 
Emergency Medical Technician (Washburn) 8.9 (6.8, 10.9) (3.4, 14.5) 
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